Tracking digital credential usage in a sensor-monitored environment

ABSTRACT

Techniques described herein relate to tracking and analyzing digital credential usage in sensor-monitored environments. A digital credential platform server may receive data identifying credential receivers, and then retrieve the digital credentials generated and issued to the credential receivers. The digital credential platform server may then determine sets of physical activities associated with the digital credentials issued to a credential receiver, and may use sensor-based monitoring systems to detect the user actions of the credential receiver and compare those actions to the physical activities associated with the credentials. Comparisons between the user actions detected for the credential receiver, and the physical activities associated with the receiver&#39;s credentials, may be used to determine re-credentialing time periods, credential expiration times, and the like.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of and claims priority to U.S.Provisional Patent Application No. 62/559,433, entitled “DIGITALCREDENTIAL PLATFORM,” filed Sep. 15, 2017, the entire contents of whichare incorporated by reference herein for all purposes.

BACKGROUND

Changes in computing technologies have provided individuals withadditional options for obtaining and validating technical skills andproficiencies. Rather than attending traditional educationalinstitutions and professional training courses, many individuals may nowobtain their technical skills and proficiencies from alternativesources, such as structured or unstructured and asynchronous eLearningprograms using distance learning technology, self-study research withoutany direct supervision, or various alternative technical learning,training, and testing entities. Although such advances in technologiesand increasing globalization trends provide many more options forindividuals to obtain technical skills and proficiencies, they alsopresent challenges in publishing, verifying, and tracking the sets oftechnical skills and proficiencies that these individuals have obtained.Many individuals and institutions no longer rely on physicalcertificates such as diplomas, transcripts, certification statements,and physical licenses, to verify the authenticity of an individual'sproficiencies or qualifications. Instead, certain institutions may issuedigital credentials (or digital badges) to qualifying individuals, andthese digital credential earners may use the digital credentials tocertify the skills or qualifications that the earner obtained vis-à-visthe institution.

BRIEF SUMMARY

Various techniques are described herein for executing and monitoringphysical simulations within a digital credential platform. In variousembodiments, techniques for generating digital credentials may includeusing physical simulation evaluation systems include sensor-basedmonitoring systems, simulation output systems, and/or simulationenvironment control systems. Within such systems, particular types ofdigital credential simulations may be determined and executed within aphysical environment, during which a plurality of sensors may be used tomonitor the physical actions of a user (or credential receiver) duringthe physical simulations. The physical action data may be analyzed todetermine particular physical activities performed by the user, and thecharacteristics of the those physical activities, such as speed,efficiency, error rate, etc. The physical activities performed may beanalyzed and compared to digital credential requirements to determineone or more digital credentials to be generated and issued to thecredential receiver, based on the monitored activities. In certainembodiments, the physical simulation may be generated by controlling oneor more output systems and/or environmental control systems.Additionally, the usage and configuration of sensors during monitoringmay be based the physical simulation being executed. Such simulationsmay include, for example, computer terminal-based physical simulations,and/or simulations requiring physical activities within the simulationenvironment.

Additional techniques are described herein for generating and issuingdigital credentials to digital credential receivers, based on actionsdetected within sensor-monitored environments. In certain embodiments, adigital credential generator may include an operation evaluation systemhaving a plurality of configurable sensors directed to detect useractivity within a physical environment. The evaluation system maymonitor a physical environment associated with a user, in order todetect various user actions performed by the user within the environmentduring a predetermined time period. Sets of user operations may bedetermined based on the physical actions, and the user operations may becompared to digital credential criteria associated with a plurality ofdifferent digital credential types. When determining that user (orcredential receiver) is eligible to receive one or more digitalcredentials based on the comparisons, the system may generate digitalcredentials based on the corresponding digital credential templates, andissue the digital credentials to the associated credential receivers. Insome examples, digital credentials based on the analyses from operationevaluation systems may include embedded credentialing time data,credentialing location data, credentialing sensor system data, and thelike. Sensors used by an operation evaluation systems may include, forexample, software-based sensors and/or video or motion detection andanalysis sensors. Additional sensors and monitoring techniques usedduring credentialing determinations may include biometric analysisand/or facial recognition for authentication, and/or credentialing.

Further techniques are described herein for tracking and analyzingdigital credential usage in sensor-monitored environments. In certainembodiments, a digital credential platform server may be configured toreceive data identifying credential receivers, and then to retrieve thedigital credentials generated and issued to those credential receivers.The digital credential platform server may then determine sets ofphysical activities associated with the digital credentials issued to acredential receiver, and may use sensor-based monitoring systems todetect the user actions of the credential receiver and compare thoseactions to the physical activities associated with the credentialsissued to that receiver. Such sensor-based monitoring systems mayinclude computer-terminal based systems and/or larger scale physicalmonitoring environments. The comparisons between the user actionsdetected for the credential receiver, and the physical activitiesassociated with the receiver's credentials, may be used to determinere-credentialing time periods, credential expiration times, etc.Additionally, in some cases, comparisons may include detecting a numberof particular action types performed by the credential receiver, errorrates, compliance with protocols, etc.

Additional techniques described herein may include generating andissuing digital credentials within a credentialing environment,including storing digital credentials with associated sensor datacollected via a sensor-monitored environment. For example, a digitalcredential generation system may include a sensor-based monitoring ordetection system, along with digital credential generation and issuingcomponents. During an evaluation of a credential receiver, andgeneration/issuance of digital credentials to the receiver, the receivermay be monitored using various sensors. The digital credentialgeneration system may determine the relevant sensor data, discardingunnecessary sensor data in some cases, and may store the relevant sensordata in a digital credential storage repository associated with theparticular credential receiver. The associated sensor data may serve asauthentication data and/or evidence of the completion of the credentialcriteria by the receiver. Additionally, in some cases, the stored sensordata may be automatically applied to additional digital credentialcriteria, such as updated criteria from the same digital credential, orcriteria for different types of digital credentials, and the system maydetermine the receiver's eligibility based on the additional criteria.Thus, such techniques may allow the digital credential generation systemto automatically generate and issue updated and/or additional digitalcredentials to receivers based on analyses of previously collectedsensor data records, rather than requiring retesting or reevaluation bythe receiver.

The techniques described herein may further include analysis,generation, and issuance of digital credentials based on feedbackassociated with credential receivers. For example, digital credentialgeneration systems may use sensor detection systems to monitor the useractions of the credential receiver and capture sensor data of the userduring the interactive assessments and other credentialing processes.Such captured sensor data may be, in some examples, non-verbal,non-written user response data detected during interactivecomputer-based assessment sessions, and may be collected using variousdevices such as biometric sensors, video or audio recording devices,facial expression or gesture capturing and analysis software, etc. Thecaptured sensor data then may be stored as feedback data, along with theresponses from the receiver collected during the interactiveassessments. Such feedback data may be evaluated and used toauthenticate the user, determine user emotion/response data duringcertain simulations and assessments. Additionally, the sensor feedbackdata collected for the user may be used in eligibility determinationsfor digital credentials in certain embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing illustrating an example of a contentdistribution network.

FIG. 2 is a block diagram illustrating a computer server and computingenvironment within a content distribution network.

FIG. 3 is a block diagram illustrating an embodiment of one or more datastore servers within a content distribution network.

FIG. 4 is a block diagram illustrating an embodiment of one or morecontent management servers within a content distribution network.

FIG. 5 is a block diagram illustrating the physical and logicalcomponents of a special-purpose computer device within a contentdistribution network.

FIG. 6 is a block diagram illustrating an example computing environmentfor generating, managing, and tracking digital credential templates anddigital credentials, according to one or more embodiments of thedisclosure.

FIG. 7 is an diagram illustrating an example computing environment forexecuting and monitoring physical simulations within a digitalcredential system, according to one or more embodiments of thedisclosure.

FIG. 8 is a flow diagram illustrating an example process of executingand monitoring physical simulations for generation of digitalcredentials, according to one or more embodiments of the disclosure.

FIG. 9A is an diagram illustrating a computer terminal-based system forsensor-based monitoring, and generation of digital credentials,according to one or more embodiments of the disclosure.

FIG. 9B is an diagram illustrating a physical environment-based systemfor sensor-based monitoring, and generation of digital credentials,according to one or more embodiments of the disclosure.

FIG. 10 is a flow diagram illustrating an example process of generatingand issuing digital credentials in a sensor-monitored environment,according to one or more embodiments of the disclosure.

FIG. 11 is an diagram illustrating an example computing environment foranalyzing sensor-based activity monitoring within a digital credentialsystem, according to one or more embodiments of the disclosure.

FIG. 12 is a flow diagram illustrating an example process of generatingdigital credentials and tracking the corresponding activities in asensor-monitored environment, according to one or more embodiments ofthe disclosure.

FIG. 13 is a flow diagram illustrating an example process of analyzingactivities in a sensor-monitored environment to determine digitalcredential expiration and/or recertification times, according to one ormore embodiments of the disclosure.

FIG. 14 is an diagram illustrating an example computing environment forgenerating and analyzing digital credentials using received sensormonitoring data, according to one or more embodiments of the disclosure.

FIG. 15 is a flow diagram illustrating an example process of generatingand storing digital credentials with associated data fromsensor-monitored environments, according to one or more embodiments ofthe disclosure.

FIGS. 16A-16B are flow diagrams illustrating example processes ofretrieving sensor data associated with issued digital credentials, andgenerating additional and/or updated digital credentials based on theretrieved sensor data, according to one or more embodiments of thedisclosure.

FIGS. 17A-17B are diagrams illustrating facial recognition and analysisfunctionality performed during digital credential generation andanalyses processes within sensor-monitored environments, according toone or more embodiments of the disclosure.

FIG. 18 is a flow diagram illustrating an example process of generatingand storing digital credentials with associated user feedback data fromsensor-monitored environments, according to one or more embodiments ofthe disclosure.

In the appended figures, similar components and/or features may have thesame reference label. Further, various compo of the same type may bedistinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

The ensuing description provides illustrative embodiment(s) only and isnot intended to limit the scope, applicability or configuration of thedisclosure. Rather, the ensuing description of the illustrativeembodiment(s) will provide those skilled in the art with an enablingdescription for implementing a preferred exemplary embodiment. It isunderstood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

Various techniques (e.g., systems, methods, computer-program productstangibly embodied in a non-transitory machine-readable storage medium,etc.) are described herein for executing and monitoring physicalsimulations within a digital credential platform. In variousembodiments, techniques for generating digital credentials may includeusing physical simulation evaluation systems include sensor-basedmonitoring systems, simulation output systems, and/or simulationenvironment control systems. Within such systems, particular types ofdigital credential simulations may be determined and executed within aphysical environment, during which a plurality of sensors may be used tomonitor the physical actions of a user (or credential receiver) duringthe physical simulations. The physical action data may be analyzed todetermine particular physical activities performed by the user, and thecharacteristics of the those physical activities, such as speed,efficiency, error rate, etc. The physical activities performed may beanalyzed and compared to digital credential requirements to determineone or more digital credentials to be generated and issued to thecredential receiver, based on the monitored activities. In certainembodiments, the physical simulation may be generated by controlling oneor more output systems and/or environmental control systems.Additionally, the usage and configuration of sensors during monitoringmay be based the physical simulation being executed. Such simulationsmay include, for example, computer terminal-based physical simulations,and/or simulations requiring physical activities within the simulationenvironment.

Additional techniques are described herein for generating and issuingdigital credentials to digital credential receivers, based on actionsdetected within sensor-monitored environments. In certain embodiments, adigital credential generator may include an operation evaluation systemhaving a plurality of configurable sensors directed to detect useractivity within a physical environment. The evaluation system maymonitor a physical environment associated with a user, in order todetect various user actions performed by the user within the environmentduring a predetermined time period. Sets of user operations may bedetermined based on the physical actions, and the user operations may becompared to digital credential criteria associated with a plurality ofdifferent digital credential types. When determining that user (orcredential receiver) is eligible to receive one or more digitalcredentials based on the comparisons, the system may generate digitalcredentials based on the corresponding digital credential templates, andissue the digital credentials to the associated credential receivers. Insome examples, digital credentials based on the analyses from operationevaluation systems may include embedded credentialing time data,credentialing location data, credentialing sensor system data, and thelike. Sensors used by an operation evaluation systems may include, forexample, software-based sensors and/or video or motion detection andanalysis sensors. Additional sensors and monitoring techniques usedduring credentialing determinations may include biometric analysisand/or facial recognition for authentication, and/or credentialing.

Further techniques are described herein for tracking and analyzingdigital credential usage in sensor-monitored environments. In certainembodiments, a digital credential platform server may be configured toreceive data identifying credential receivers, and then to retrieve thedigital credentials generated and issued to those credential receivers.The digital credential platform server may then determine sets ofphysical activities associated with the digital credentials issued to acredential receiver, and may use sensor-based monitoring systems todetect the user actions of the credential receiver and compare thoseactions to the physical activities associated with the credentialsissued to that receiver. Such sensor-based monitoring systems mayinclude computer-terminal based systems and/or larger scale physicalmonitoring environments. The comparisons between the user actionsdetected for the credential receiver, and the physical activitiesassociated with the receiver's credentials, may be used to determinere-credentialing time periods, credential expiration times, etc.Additionally, in some cases, comparisons may include detecting a numberof particular action types performed by the credential receiver, errorrates, compliance with protocols, etc.

Additional techniques described herein may include generating andissuing digital credentials within a credentialing environment,including storing digital credentials with associated sensor datacollected via a sensor-monitored environment. For example, a digitalcredential generation system may include a sensor-based monitoring ordetection system, along with digital credential generation and issuingcomponents. During an evaluation of a credential receiver, andgeneration/issuance of digital credentials to the receiver, the receivermay be monitored using various sensors. The digital credentialgeneration system may determine the relevant sensor data, discardingunnecessary sensor data in some cases, and may store the relevant sensordata in a digital credential storage repository associated with theparticular credential receiver. The associated sensor data may serve asauthentication data and/or evidence of the completion of the credentialcriteria by the receiver. Additionally, in some cases, the stored sensordata may be automatically applied to additional digital credentialcriteria, such as updated criteria from the same digital credential, orcriteria for different types of digital credentials, and the system maydetermine the receiver's eligibility based on the additional criteria.Thus, such techniques may allow the digital credential generation systemto automatically generate and issue updated and/or additional digitalcredentials to receivers based on analyses of previously collectedsensor data records, rather than requiring retesting or reevaluation bythe receiver.

The techniques described herein may further include analysis,generation, and issuance of digital credentials based on feedbackassociated with credential receivers. For example, digital credentialgeneration systems may use sensor detection systems to monitor the useractions of the credential receiver and capture sensor data of the userduring the interactive assessments and other credentialing processes.Such captured sensor data may be, in some examples, non-verbal,non-written user response data detected during interactivecomputer-based assessment sessions, and may be collected using variousdevices such as biometric sensors, video or audio recording devices,facial expression or gesture capturing and analysis software, etc. Thecaptured sensor data then may be stored as feedback data, along with theresponses from the receiver collected during the interactiveassessments. Such feedback data may be evaluated and used toauthenticate the user, determine user emotion/response data duringcertain simulations and assessments. Additionally, the sensor feedbackdata collected for the user may be used in eligibility determinationsfor digital credentials in certain embodiments.

With reference now to FIG. 1, a block diagram is shown illustratingvarious components of a content distribution network (CDN) 100 whichimplements and supports certain embodiments and features describedherein. Content distribution network 100 may include one or more contentmanagement servers 102. As discussed below in more detail, contentmanagement servers 102 may be any desired type of server including, forexample, a rack server, a tower server, a miniature server, a bladeserver, a mini rack server, a mobile server, an ultra-dense server, asuper server, or the like, and may include various hardware components,for example, a motherboard, a processing units, memory systems, harddrives, network interfaces, power supplies, etc. Content managementserver 102 may include one or more server farms, clusters, or any otherappropriate arrangement and/or combination or computer servers. Contentmanagement server 102 may act according to stored instructions locatedin a memory subsystem of the server 102, and may run an operatingsystem, including any commercially available server operating systemand/or any other operating systems discussed herein.

The content distribution network 100 may include one or more data storeservers 104, such as database servers and file-based storage systems.Data stores 104 may comprise stored data relevant to the functions ofthe content distribution network 100. Illustrative examples of datastores 104 that may be maintained in certain embodiments of the contentdistribution network 100 are described below in reference to FIG. 3. Insome embodiments, multiple data stores may reside on a single server104, either using the same storage components of server 104 or usingdifferent physical storage components to assure data security andintegrity between data stores. In other embodiments, each data store mayhave a separate dedicated data store server 104.

Content distribution network 100 also may include one or more userdevices 106 and/or supervisor devices 110. User devices 106 andsupervisor devices 110 may display content received via the contentdistribution network 100, and may support various types of userinteractions with the content. User devices 106 and supervisor devices110 may include mobile devices such as smartphones, tablet computers,personal digital assistants, and wearable computing devices. Such mobiledevices may run a variety of mobile operating systems, and may beenabled for Internet, e-mail, short message service (SMS), Bluetooth®,mobile radio-frequency identification (M-RFID), and/or othercommunication protocols. Other user devices 106 and supervisor devices110 may be general purpose personal computers or special-purposecomputing devices including, by way of example, personal computers,laptop computers, workstation computers, projection devices, andinteractive room display systems. Additionally, user devices 106 andsupervisor devices 110 may be any other electronic devices, such asthin-client computers, Internet-enabled gaming systems, business or homeappliances, and/or personal messaging devices, capable of communicatingover network(s) 120.

In different contexts of content distribution networks 100, user devices106 and supervisor devices 110 may correspond to different types ofspecialized devices, for example, student devices and teacher devices inan educational network, employee devices and presentation devices in acompany network, different gaming devices in a gaming network, etc. Insome embodiments, user devices 106 and supervisor devices 110 mayoperate in the same physical location 107, such as a classroom orconference room. In such cases, the devices may contain components thatsupport direct communications with other nearby devices, such as awireless transceivers and wireless communications interfaces, Ethernetsockets or other Local Area Network (LAN) interfaces, etc. In otherimplementations, the user devices 106 and supervisor devices 110 neednot be used at the same location 107, but may be used in remotegeographic locations in which each user device 106 and supervisor device110 may use security features and/or specialized hardware (e.g.,hardware-accelerated SSL and HTTPS, WS-Security, firewalls, etc.) tocommunicate with the content management server 102 and/or other remotelylocated user devices 106. Additionally, different user devices 106 andsupervisor devices 110 may be assigned different designated roles, suchas presenter devices, teacher devices, administrator devices, or thelike, and in such cases the different devices may be provided withadditional hardware and/or software components to provide content andsupport user capabilities not available to the other devices.

The content distribution network 100 also may include a privacy server108 that maintains private user information at the privacy server 108while using applications or services hosted on other servers. Forexample, the privacy server 108 may be used to maintain private data ofa user within one jurisdiction even though the user is accessing anapplication hosted on a server (e.g., the content management server 102)located outside the jurisdiction. In such cases, the privacy server 108may intercept communications between a user device 106 or supervisordevice 110 and other devices that include private user information. Theprivacy server 108 may create a token or identifier that does notdisclose the private information and may use the token or identifierwhen communicating with the other servers and systems, instead of usingthe user's private information.

As illustrated in FIG. 1, the content management server 102 may be incommunication with one or more additional servers, such as a contentserver 112, a user data server 112, and/or an administrator server 116.Each of these servers may include some or all of the same physical andlogical components as the content management server(s) 102, and in somecases, the hardware and software components of these servers 112-116 maybe incorporated into the content management server(s) 102, rather thanbeing implemented as separate computer servers.

Content server 112 may include hardware and software components togenerate, store, and maintain the content resources for distribution touser devices 106 and other devices in the network 100. For example, incontent distribution networks 100 used for professional training andeducational purposes, content server 112 may include data stores oftraining materials, presentations, interactive programs and simulations,course models, course outlines, and various training interfaces thatcorrespond to different materials and/or different types of user devices106. In content distribution networks 100 used for media distribution,interactive gaming, and the like, a content server 112 may include mediacontent files such as music, movies, television programming, games, andadvertisements.

User data server 114 may include hardware and software components thatstore and process data for multiple users relating to each user'sactivities and usage of the content distribution network 100. Forexample, the content management server 102 may record and track eachuser's system usage, including their user device 106, content resourcesaccessed, and interactions with other user devices 106. This data may bestored and processed by the user data server 114, to support usertracking and analysis features. For instance, in the professionaltraining and educational contexts, the user data server 114 may storeand analyze each user's training materials viewed, presentationsattended, courses completed, interactions, evaluation results, and thelike. The user data server 114 may also include a repository foruser-generated material, such as evaluations and tests completed byusers, and documents and assignments prepared by users. In the contextof media distribution and interactive gaming, the user data server 114may store and process resource access data for multiple users (e.g.,content titles accessed, access times, data usage amounts, gaminghistories, user devices and device types, etc.).

Administrator server 116 may include hardware and software components toinitiate various administrative functions at the content managementserver 102 and other components within the content distribution network100. For example, the administrator server 116 may monitor device statusand performance for the various servers, data stores, and/or userdevices 106 in the content distribution network 100. When necessary, theadministrator server 116 may add or remove devices from the network 100,and perform device maintenance such as providing software updates to thedevices in the network 100. Various administrative tools on theadministrator server 116 may allow authorized users to set user accesspermissions to various content resources, monitor resource usage byusers and devices 106, and perform analyses and generate reports onspecific network users and/or devices (e.g., resource usage trackingreports, training evaluations, etc.).

The content distribution network 100 may include one or morecommunication networks 120. Although only a single network 120 isidentified in FIG. 1, the content distribution network 100 may includeany number of different communication networks between any of thecomputer servers and devices shown in FIG. 1 and/or other devicesdescribed herein. Communication networks 120 may enable communicationbetween the various computing devices, servers, and other components ofthe content distribution network 100. As discussed below, variousimplementations of content distribution networks 100 may employdifferent types of networks 120, for example, computer networks,telecommunications networks, wireless networks, and/or any combinationof these and/or other networks.

With reference to FIG. 2, an illustrative distributed computingenvironment 200 is shown including a computer server 202, four clientcomputing devices 206, and other components that may implement certainembodiments and features described herein. In some embodiments, theserver 202 may correspond to the content management server 102 discussedabove in FIG. 1, and the client computing devices 206 may correspond tothe user devices 106. However, the computing environment 200 illustratedin FIG. 2 may correspond to any other combination of devices and serversconfigured to implement a client-server model or other distributedcomputing architecture.

Client devices 206 may be configured to receive and execute clientapplications over one or more networks 220. Such client applications maybe web browser based applications and/or standalone softwareapplications, such as mobile device applications. Server 202 may becommunicatively coupled with the client devices 206 via one or morecommunication networks 220. Client devices 206 may receive clientapplications from server 202 or from other application providers (e.g.,public or private application stores). Server 202 may be configured torun one or more server software applications or services, for example,web-based or cloud-based services, to support content distribution andinteraction with client devices 206. Users operating client devices 206may in turn utilize one or more client applications (e.g., virtualclient applications) to interact with server 202 to utilize the servicesprovided by these components.

Various different subsystems and/or components 204 may be implemented onserver 202. Users operating the client devices 206 may initiate one ormore client applications to use services provided by these subsystemsand components. The subsystems and components within the server 202 andclient devices 206 may be implemented in hardware, firmware, software,or combinations thereof. Various different system configurations arepossible in different distributed computing systems 200 and contentdistribution networks 100. The embodiment shown in FIG. 2 is thus oneexample of a distributed computing system and is not intended to belimiting.

Although exemplary computing environment 200 is shown with four clientcomputing devices 206, any number of client computing devices may besupported. Other devices, such as specialized sensor devices, etc., mayinteract with client devices 206 and/or server 202.

As shown in FIG. 2, various security and integration components 208 maybe used to transmit, receive, and manage communications between theserver 202 and user devices 206 over one or more communication networks220. The security and integration components 208 may include separateservers, such as web servers and/or authentication servers, and/orspecialized networking components, such as firewalls, routers, gateways,load balancers, and the like. In some cases, the security andintegration components 208 may correspond to a set of dedicated hardwareand/or software operating at the same physical location and under thecontrol of same entities as server 202. For example, components 208 mayinclude one or more dedicated web servers and network hardware in adatacenter or a cloud infrastructure. In other examples, the securityand integration components 208 may correspond to separate hardware andsoftware components which may be operated at a separate physicallocation and/or by a separate entity.

Security and integration components 208 may implement various securityfeatures for data transmission and storage, such as authenticating usersand restricting access to unknown or unauthorized users. In variousimplementations, security and integration components 208 may provide,for example, a file-based integration scheme or a service-basedintegration scheme for transmitting data between the various devices inthe content distribution network 100. Security and integrationcomponents 208 also may use secure data transmission protocols and/orencryption for data transfers, for example, File Transfer Protocol(FTP), Secure File Transfer Protocol (SFTP), and/or Pretty Good Privacy(PGP) encryption.

In some embodiments, one or more web services may be implemented withinthe security and integration components 208 and/or elsewhere within thecontent distribution network 100. Such web services, includingcross-domain and/or cross-platform web services, may be developed forenterprise use in accordance with various web service standards, such asRESTful web services (i.e., services based on the Representation StateTransfer (REST) architectural style and constraints), and/or webservices designed in accordance with the Web Service Interoperability(WS-I) guidelines. Some web services may use the Secure Sockets Layer(SSL) or Transport Layer Security (TLS) protocol to provide secureconnections between the server 202 and user devices 206. SSL or TLS mayuse HTTP or HTTPS to provide authentication and confidentiality. Inother examples, web services may be implemented using REST over HTTPSwith the OAuth open standard for authentication, or using theWS-Security standard which provides for secure SOAP messages using XML,encryption. In other examples, the security and integration components208 may include specialized hardware for providing secure web services.For example, security and integration components 208 may include securenetwork appliances having built-in features such as hardware-acceleratedSSL and HTTPS, WS-Security, and firewalls. Such specialized hardware maybe installed and configured in front of any web servers, so that anyexternal devices may communicate directly with the specialized hardware.

Communication network(s) 220 may be any type of network familiar tothose skilled in the art that can support data communications using anyof a variety of commercially-available protocols, including withoutlimitation, TCP/IP (transmission control protocol/Internet protocol),SNA (systems network architecture), IPX (Internet packet exchange),Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols,Hyper Text Transfer Protocol (HTTP) and Secure Hyper Text TransferProtocol (HTTPS), Bluetooth®, Near Field Communication (NFC), and thelike. Merely by way of example, network(s) 220 may be local areanetworks (LAN), such as one based on Ethernet, Token-Ring and/or thelike. Network(s) 220 also may be wide-area networks, such as theInternet. Networks 220 may include telecommunication networks such as apublic switched telephone networks (PSTNs), or virtual networks such asan intranet or an extranet. Infrared and wireless networks (e.g., usingthe Institute of Electrical and Electronics (IEEE) 802.11 protocol suiteor other wireless protocols) also may be included in networks 220.

Computing environment 200 also may include one or more data stores 210and/or back-end servers 212. In certain examples, the data stores 210may correspond to data store server(s) 104 discussed above in FIG. 1,and back-end servers 212 may correspond to the various back-end servers112-116. Data stores 210 and servers 212 may reside in the samedatacenter or may operate at a remote location from server 202. In somecases, one or more data stores 210 may reside on a non-transitorystorage medium within the server 202. Other data stores 210 and back-endservers 212 may be remote from server 202 and configured to communicatewith server 202 via one or more networks 220. In certain embodiments,data stores 210 and back-end servers 212 may reside in a storage-areanetwork (SAN), or may use storage-as-a-service (STaaS) architecturalmodel.

With reference to FIG. 3, an illustrative set of data stores and/or datastore servers is shown, corresponding to the data store servers 104 ofthe content distribution network 100 discussed above in FIG. 1. One ormore individual data stores 301-309 may reside in storage on a singlecomputer server 104 (or a single server farm or cluster) under thecontrol of a single entity, or may reside on separate servers operatedby different entities and/or at remote locations. In some embodiments,data stores 301-309 may be accessed by the content management server 102and/or other devices and servers within the network 100 (e.g., userdevices 106, supervisor devices 110, administrator servers 116, etc.).Access to one or more of the data stores 301-309 may be limited ordenied based on the processes, user credentials, and/or devicesattempting to interact with the data store.

The paragraphs below describe examples of specific data stores that maybe implemented within some embodiments of a content distribution network100. It should be understood that the below descriptions of data stores301-309, including their functionality and types of data stored therein,are illustrative and non-limiting. Data stores server architecture,design, and the execution of specific data stores 301-309 may depend onthe context, size, and functional requirements of a content distributionnetwork 100. For example, in content distribution systems 100 used forprofessional training and educational purposes, separate databases orfile-based storage systems may be implemented in data store server(s)104 to store trainee and/or student data, trainer and/or professor data,training module data and content descriptions, training results,evaluation data, and the like. In contrast, in content distributionsystems 100 used for media distribution from content providers tosubscribers, separate data stores may be implemented in data storesserver(s) 104 to store listings of available content titles anddescriptions, content title usage statistics, subscriber profiles,account data, payment data, network usage statistics, etc.

A user profile data store 301 may include information relating to theend users within the content distribution network 100. This informationmay include user characteristics such as the user names, accesscredentials (e.g., logins and passwords), user preferences, andinformation relating to any previous user interactions within thecontent distribution network 100 (e.g., requested content, postedcontent, content modules completed, training scores or evaluations,other associated users, etc.).

An accounts data store 302 may generate and store account data fordifferent users in various roles within the content distribution network100. For example, accounts may be created in an accounts data store 302for individual end users, supervisors, administrator users, and entitiessuch as companies or educational institutions. Account data may includeaccount types, current account status, account characteristics, and anyparameters, limits, restrictions associated with the accounts.

A content library data store 303 may include information describing theindividual content items (or content resources) available via thecontent distribution network 100. In some embodiments, the library datastore 303 may include metadata, properties, and other characteristicsassociated with the content resources stored in the content server 112.Such data may identify one or more aspects or content attributes of theassociated content resources, for example, subject matter, access level,or skill level of the content resources, license attributes of thecontent resources (e.g., any limitations and/or restrictions on thelicensable use and/or distribution of the content resource), priceattributes of the content resources (e.g., a price and/or pricestructure for determining a payment amount for use or distribution ofthe content resource), rating attributes for the content resources(e.g., data indicating the evaluation or effectiveness of the contentresource), and the like. In some embodiments, the library data store 303may be configured to allow updating of content metadata or properties,and to allow the addition and/or removal of information relating to thecontent resources. For example, content relationships may be implementedas graph structures, which may be stored in the library data store 303or in an additional store for use by selection algorithms along with theother metadata.

A pricing data store 304 may include pricing information and/or pricingstructures for determining payment amounts for providing access to thecontent distribution network 100 and/or the individual content resourceswithin the network 100. In some cases, pricing may be determined basedon a user's access to the content distribution network 100, for example,a time-based subscription fee, or pricing based on network usage and. Inother cases, pricing may be tied to specific content resources. Certaincontent resources may have associated pricing information, whereas otherpricing determinations may be based on the resources accessed, theprofiles and/or accounts of the user, and the desired level of access(e.g., duration of access, network speed, etc.). Additionally, thepricing data store 304 may include information relating to compilationpricing for groups of content resources, such as group prices and/orprice structures for groupings of resources.

A license data store 305 may include information relating to licensesand/or licensing of the content resources within the contentdistribution network 100. For example, the license data store 305 mayidentify licenses and licensing terms for individual content resourcesand/or compilations of content resources in the content server 112, therights holders for the content resources, and/or common or large-scaleright holder information such as contact information for rights holdersof content not included in the content server 112.

A content access data store 306 may include access rights and securityinformation for the content distribution network 100 and specificcontent resources. For example, the content access data store 306 mayinclude login information (e.g., user identifiers, logins, passwords,etc.) that can be verified during user login attempts to the network100. The content access data store 306 also may be used to storeassigned user roles and/or user levels of access. For example, a user'saccess level may correspond to the sets of content resources and/or theclient or server applications that the user is permitted to access.Certain users may be permitted or denied access to certain applicationsand resources based on their subscription level, training program,course/grade level, etc. Certain users may have supervisory access overone or more end users, allowing the supervisor to access all or portionsof the end user's content, activities, evaluations, etc. Additionally,certain users may have administrative access over some users and/or someapplications in the content management network 100, allowing such usersto add and remove user accounts, modify user access permissions, performmaintenance updates on software and servers, etc.

A source data store 307 may include information relating to the sourceof the content resources available via the content distribution network.For example, a source data store 307 may identify the authors andoriginating devices of content resources, previous pieces of data and/orgroups of data originating from the same authors or originating devices,and the like.

An evaluation data store 308 may include information used to direct theevaluation of users and content resources in the content managementnetwork 100. In some embodiments, the evaluation data store 308 maycontain, for example, the analysis criteria and the analysis guidelinesfor evaluating users (e.g., trainees/students, gaming users, mediacontent consumers, etc.) and/or for evaluating the content resources inthe network 100. The evaluation data store 308 also may includeinformation relating to evaluation processing tasks, for example, theidentification of users and user devices 106 that have received certaincontent resources or accessed certain applications, the status ofevaluations or evaluation histories for content resources, users, orapplications, and the like. Evaluation criteria may be stored in theevaluation data store 308 including data and/or instructions in the formof one or several electronic rubrics or scoring guides for use in theevaluation of the content, users, or applications. The evaluation datastore 308 also may include past evaluations and/or evaluation analysesfor users, content, and applications, including relative rankings,characterizations, explanations, and the like.

In addition to the illustrative data stores described above, data storeserver(s) 104 (e.g., database servers, file-based storage servers, etc.)may include one or more external data aggregators 309. External dataaggregators 309 may include third-party data sources accessible to thecontent management network 100, but not maintained by the contentmanagement network 100. External data aggregators 309 may include anyelectronic information source relating to the users, content resources,or applications of the content distribution network 100. For example,external data aggregators 309 may be third-party data stores containingdemographic data, education related data, consumer sales data, healthrelated data, and the like. Illustrative external data aggregators 309may include, for example, social networking web servers, public recordsdata stores, learning management systems, educational institutionservers, business servers, consumer sales data stores, medical recorddata stores, etc. Data retrieved from various external data aggregators309 may be used to verify and update user account information, suggestuser content, and perform user and content evaluations.

With reference now to FIG. 4, a block diagram is shown illustrating anembodiment of one or more content management servers 102 within acontent distribution network 100. As discussed above, content managementserver(s) 102 may include various server hardware and softwarecomponents that manage the content resources within the contentdistribution network 100 and provide interactive and adaptive content tousers on various user devices 106. For example, content managementserver(s) 102 may provide instructions to and receive information fromthe other devices within the content distribution network 100, in orderto manage and transmit content resources, user data, and server orclient applications executing within the network 100.

A content management server 102 may include a content customizationsystem 402. The content customization system 402 may be implementedusing dedicated hardware within the content distribution network 100(e.g., a content customization server 402), or using designated hardwareand software resources within a shared content management server 102. Insome embodiments, the content customization system 402 may adjust theselection and adaptive capabilities of content resources to match theneeds and desires of the users receiving the content. For example, thecontent customization system 402 may query various data stores andservers 104 to retrieve user information, such as user preferences andcharacteristics (e.g., from a user profile data store 301), user accessrestrictions to content recourses (e.g., from a content access datastore 306), previous user results and content evaluations (e.g., from anevaluation data store 308), and the like. Based on the retrievedinformation from data stores 104 and other data sources, the contentcustomization system 402 may modify content resources for individualusers.

A content management server 102 also may include a user managementsystem 404. The user management system 404 may be implemented usingdedicated hardware within the content distribution network 100 (e.g., auser management server 404), or using designated hardware and softwareresources within a shared content management server 102. In someembodiments, the user management system 404 may monitor the progress ofusers through various types of content resources and groups, such asmedia compilations, courses or curriculums in training or educationalcontexts, interactive gaming environments, and the like. For example,the user management system 404 may query one or more databases and/ordata store servers 104 to retrieve user data such as associated contentcompilations or programs, content completion status, user goals,results, and the like.

A content management server 102 also may include an evaluation system406. The evaluation system 406 may be implemented using dedicatedhardware within the content distribution network 100 (e.g., anevaluation server 406), or using designated hardware and softwareresources within a shared content management server 102. The evaluationsystem 406 may be configured to receive and analyze information fromuser devices 106. For example, various ratings of content resourcessubmitted by users may be compiled and analyzed, and then stored in adata store (e.g., a content library data store 303 and/or evaluationdata store 308) associated with the content. In some embodiments, theevaluation server 406 may analyze the information to determine theeffectiveness or appropriateness of content resources with, for example,a subject matter, an age group, a skill level, or the like. In someembodiments, the evaluation system 406 may provide updates to thecontent customization system 402 or the user management system 404, withthe attributes of one or more content resources or groups of resourceswithin the network 100. The evaluation system 406 also may receive andanalyze user evaluation data from user devices 106, supervisor devices110, and administrator servers 116, etc. For instance, evaluation system406 may receive, aggregate, and analyze user evaluation data fordifferent types of users (e.g., end users, supervisors, administrators,etc.) in different contexts (e.g., media consumer ratings, trainee orstudent comprehension levels, teacher effectiveness levels, gamer skilllevels, etc.).

A content management server 102 also may include a content deliverysystem 408. The content delivery system 408 may be implemented usingdedicated hardware within the content distribution network 100 (e.g., acontent delivery server 408), or using designated hardware and softwareresources within a shared content management server 102. The contentdelivery system 408 may receive content resources from the contentcustomization system 402 and/or from the user management system 404, andprovide the resources to user devices 106. The content delivery system408 may determine the appropriate presentation format for the contentresources based on the user characteristics and preferences, and/or thedevice capabilities of user devices 106. If needed, the content deliverysystem 408 may convert the content resources to the appropriatepresentation format and/or compress the content before transmission. Insome embodiments, the content delivery system 408 may also determine theappropriate transmission media and communication protocols fortransmission of the content resources.

In some embodiments, the content delivery system 408 may includespecialized security and integration hardware 410, along withcorresponding software components to implement the appropriate securityfeatures content transmission and storage, to provide the supportednetwork and client access models, and to support the performance andscalability requirements of the network 100. The security andintegration layer 410 may include some or all of the security andintegration components 208 discussed above in FIG. 2, and may controlthe transmission of content resources and other data, as well as thereceipt of requests and content interactions, to and from the userdevices 106, supervisor devices 110, administrative servers 116, andother devices in the network 100.

With reference now to FIG. 5, a block diagram of an illustrativecomputer system is shown. The system 500 may correspond to any of thecomputing devices or servers of the content distribution network 100described above, or any other computing devices described herein. Inthis example, computer system 500 includes processing units 504 thatcommunicate with a number of peripheral subsystems via a bus subsystem502. These peripheral subsystems include, for example, a storagesubsystem 510, an I/O subsystem 526, and a communications subsystem 532.

Bus subsystem 502 provides a mechanism for letting the variouscomponents and subsystems of computer system 500 communicate with eachother as intended. Although bus subsystem 502 is shown schematically asa single bus, alternative embodiments of the bus subsystem may utilizemultiple buses. Bus subsystem 502 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Sucharchitectures may include, for example, an Industry

Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus,Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA)local bus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 504, which may be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 500. One or more processors,including single core and/or multicore processors, may be included inprocessing unit 504. As shown in the figure, processing unit 504 may beimplemented as one or more independent processing units 506 and/or 508with single or multicore processors and processor caches included ineach processing unit. In other embodiments, processing unit 504 may alsobe implemented as a quad-core processing unit or larger multicoredesigns (e.g., hexa-core processors, octo-core processors, ten-coreprocessors, or greater.

Processing unit 504 may execute a variety of software processes embodiedin program code, and may maintain multiple concurrently executingprograms or processes. At any given time, some or all of the programcode to be executed can be resident in processor(s) 504 and/or instorage subsystem 510. In some embodiments, computer system 500 mayinclude one or more specialized processors, such as digital signalprocessors (DSPs), outboard processors, graphics processors,application-specific processors, and/or the like.

I/O subsystem 526 may include device controllers 528 for one or moreuser interface input devices and/or user interface output devices 530.User interface input and output devices 530 may be integral with thecomputer system 500 (e.g., integrated audio/video systems, and/ortouchscreen displays), or may be separate peripheral devices which areattachable/detachable from the computer system 500.

Input devices 530 may include a keyboard, pointing devices such as amouse or trackball, a touchpad or touch screen incorporated into adisplay, a scroll wheel, a click wheel, a dial, a button, a switch, akeypad, audio input devices with voice command recognition systems,microphones, and other types of input devices. Input devices 530 mayalso include three dimensional (3D) mice, joysticks or pointing sticks,gamepads and graphic tablets, and audio/visual devices such as speakers,digital cameras, digital camcorders, portable media players, webcams,image scanners, fingerprint scanners, barcode reader 3D scanners, 3Dprinters, laser rangefinders, and eye gaze tracking devices. Additionalinput devices 530 may include, for example, motion sensing and/orgesture recognition devices that enable users to control and interactwith an input device through a natural user interface using gestures andspoken commands, eye gesture recognition devices that detect eyeactivity from users and transform the eye gestures as input into aninput device, voice recognition sensing devices that enable users tointeract with voice recognition systems through voice commands, medicalimaging input devices, MIDI keyboards, digital musical instruments, andthe like.

Output devices 530 may include one or more display subsystems, indicatorlights, or non-visual displays such as audio output devices, etc.Display subsystems may include, for example, cathode ray tube (CRT)displays, flat-panel devices, such as those using a liquid crystaldisplay (LCD) or plasma display, light-emitting diode (LED) displays,projection devices, touch screens, and the like. In general, use of theterm “output device” is intended to include all possible types ofdevices and mechanisms for outputting information from computer system500 to a user or other computer. For example, output devices 530 mayinclude, without limitation, a variety of display devices that visuallyconvey text, graphics and audio/video information such as monitors,printers, speakers, headphones, automotive navigation systems, plotters,voice output devices, and modems.

Computer system 500 may comprise one or more storage subsystems 510,comprising hardware and software components used for storing data andprogram instructions, such as system memory 518 and computer-readablestorage media 516. The system memory 518 and/or computer-readablestorage media 516 may store program instructions that are loadable andexecutable on processing units 504, as well as data generated during theexecution of these programs.

Depending on the configuration and type of computer system 500, systemmemory 318 may be stored in volatile memory (such as random accessmemory (RAM) 512) and/or in non-volatile storage drives 514 (such asread-only memory (ROM), flash memory, etc.) The RAM 512 may contain dataand/or program modules that are immediately accessible to and/orpresently being operated and executed by processing units 504. In someimplementations, system memory 518 may include multiple different typesof memory, such as static random access memory (SRAM) or dynamic randomaccess memory (DRAM). In some implementations, a basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within computer system 500, such as duringstart-up, may typically be stored in the non-volatile storage drives514. By way of example, and not limitation, system memory 518 mayinclude application programs 520, such as client applications, Webbrowsers, mid-tier applications, server applications, etc., program data522, and an operating system 524.

Storage subsystem 510 also may provide one or more tangiblecomputer-readable storage media 516 for storing the basic programmingand data constructs that provide the functionality of some embodiments.Software (programs, code modules, instructions) that when executed by aprocessor provide the functionality described herein may be stored instorage subsystem 510. These software modules or instructions may beexecuted by processing units 504. Storage subsystem 510 may also providea repository for storing data used in accordance with the presentinvention.

Storage subsystem 300 may also include a computer-readable storage mediareader that can further be connected to computer-readable storage media516. Together and, optionally, in combination with system memory 518,computer-readable storage media 516 may comprehensively representremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containing, storing,transmitting, and retrieving computer-readable information.

Computer-readable storage media 516 containing program code, or portionsof program code, may include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computer system 500.

By way of example, computer-readable storage media 516 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 516 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 516 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 500.

Communications subsystem 532 may provide a communication interface fromcomputer system 500 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 5, thecommunications subsystem 532 may include, for example, one or morenetwork interface controllers (NICs) 534, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 536, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally and/or alternatively, the communicationssubsystem 532 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, FireWire® interfaces, USB® interfaces, and the like.Communications subsystem 536 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

The various physical components of the communications subsystem 532 maybe detachable components coupled to the computer system 500 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 500. Communicationssubsystem 532 also may be implemented in whole or in part by software.

In some embodiments, communications subsystem 532 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 500. For example,communications subsystem 532 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators 309). Additionally, communications subsystem 532may be configured to receive data in the form of continuous datastreams, which may include event streams of real-time events and/orevent updates (e.g., sensor data applications, financial tickers,network performance measuring tools, clickstream analysis tools,automobile traffic monitoring, etc.). Communications subsystem 532 mayoutput such structured and/or unstructured data feeds, event streams,event updates, and the like to one or more data stores 104 that may bein communication with one or more streaming data source computerscoupled to computer system 500.

Due to the ever-changing nature of computers and networks, thedescription of computer system 500 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

With reference now to FIG. 6, a block diagram is shown illustrating anexample of a digital credential management system 600 for generating,managing, and tracking digital credential templates and digitalcredentials. As shown in this example, a digital credential managementsystem 600 may include a digital credential platform server 610configured to communicate with various other digital credential systems620-680. As discussed below, the digital credential platform server 610may receive and store digital credential templates from various digitalcredential template owner systems 620. Systems 620 may correspond to thecomputer servers and/or devices of educational institutions orprofessional training organizations, which may have the primaryresponsibility for defining a digital credential template andcontrolling the content and requirements for users to receive a digitalcredential from the organization. The digital credential managementsystem 600 may include one or more digital credential issuer systems630. As discussed below, each issuer system 630 may communicate with theplatform server to request and receive access to issue digitalcredentials based on specific digital credential templates. The platformserver 610 may process template access requests from the credentialissuer systems 630, permitting or denying a specific system 630 togenerate (or issue) a digital credential based on a specific digitalcredential template.

As used herein, a digital credential template (or digital badgetemplate) may refer to an electronic document or data structure storinga general (e.g., non-user specific) template or description of aspecific type of digital credential that may be issued to an individual.Digital credential templates may include, for example, a description ofthe skills, proficiencies, and/or achievements that the digitalcredential represents. This description may take the form of diplomadata, certification data, and/or license data, including the parentorganization (i.e., the digital credential template owner) responsiblefor creating and defining the digital credential template. Examples ofdigital credential templates may include templates for varioustechnology certifications, licensure exams, professional tests, trainingcourse completion certificates, and the like. In contrast to a digitalcredential template, a digital credential (or digital badge) may referto an instance of an electronic document or data structure, generatedfor a specific individual (i.e., the credential receiver), and based ona digital credential template. Thus, a digital credential document ordata structure may be based on a corresponding digital credentialtemplate, but may be customized and populated with user-specificinformation such as individual identification data (e.g., name, emailaddress, and other user identifiers), credential issuance data (e.g.,issue date, geographic location of issuance, authorized issuer of thecredential, etc.), and links or embedded data that contain the specificuser's supporting documentation or evidence relating to the credential.

As shown in this example, the system 600 also may include a digitalcredential receiver system 640 and a digital credential endorser system650. The digital credential receiver system 640 may be a computingdevice associated with a credential receiver (or credential earner), forexample, an individual user of an electronic learning system,professional training system, online certification course, etc. In someembodiments, credential receivers may access the platform server 610 viasystems 640 to accept or reject newly issued digital credentials, reviewand update their own set of previously earned digital credentials, aswell as to publish (or share) their digital credentials viacommunication applications or publishing platforms such as social mediasystems. Digital credential endorser system 650 may be a computingsystem associated with an endorsing entity, such as an educationalinstitution, business, or technical organization that has chosen toreview and endorse a specific digital credential template. The platformserver 610 may receive and track the endorsements received from systems650, and may associate the endorsements with the user-specific digitalcredentials issued based on the endorsed templates.

Additionally, the digital credential management system 600 in thisexample includes a number of external client devices 660 and externaldigital credential publishers 670. External client devices 660 maycorrespond to computing systems of third-party users that may interactwith the platform server 610 to initiate various functionality orretrieve data relating to templates and/digital credentials managed bythe platform 610. For example, a client device 660 may query theplatform server 610 for data metrics and/or analyses relating to asubset of digital credentials stored in the digital credential datastore 615. The third-party systems 660 also may provide data to theplatform server 610 that may initiate updates to the templatesand/digital credentials stored in the data store 615. External digitalcredential publishers 670 may correspond to third-party systemsconfigured to receive digital credential data from the platform 610 andpublish (or present) the digital credential data to users. Examples ofpublishers 670 may include social media website and systems, digitalbadge wallets, and/or other specialized servers or applicationsconfigured to store and present views of digital badges to users.

In various embodiments described herein, the generation and managementof digital credentials, as well as the tracking and reporting of digitalcredential data, may be performed within CDNs 100, such as eLearning,professional training, and certification systems 100. For example,within the context of an eLearning CDN 100, a content management server102 or other CDN server (e.g., 104, 112, 114, or 116) may create andstore digital credential templates to describe and define variousproficiencies, achievements, or certifications supported by theeLearning CDN 100. Additionally or alternatively, the content managementserver 102 or other servers of an eLearning CDN 100 may issue digitalcredentials to users, based on its own digital certificate templatesand/or templates received from other systems or CDNs. Further, in someimplementations, an eLearning CDN 100 may be configured to include adigital credential platform server 610 to store and manage templates anddigital credentials between separate systems within the CDN 100. Thus,in various different implementations, the content management server(s)102 of a CDN 100 may incorporate one or more digital certificatetemplate owner system(s) 620, digital certificate issuer system(s) 630,and/or digital certificate platform server(s) 610. In such embodiments,the various components and functionalities described herein for theplatform server 610, owner system 620, and/or issuer system 630 all maybe implemented within one or more content management servers 102 (and/orother servers) of an eLearning or professional training CDN 100. Inother examples, a digital credential platform server 610 may beimplemented using one or more computer servers, and other specializedhardware and software components, separately from any other CDNcomponents such as content servers 112, content management servers 102,data store servers 104, and the like. In these examples, the digitalcredential platform server 610 may be configured to communicate directlywith related systems 620-670, or indirectly through content managementservers 102 and/or other components and communications networks of theCDN 100.

In order to perform these features and other functionality describedherein, each of the components and sub-components discussed in theexample digital credential management system 600 may correspond to asingle computer server or a complex computing system including acombination of computing devices, storage devices, network components,etc. Each of these components and their respective subcomponents may beimplemented in hardware, software, or a combination thereof. Certainsystems 620-670 may communicate directly with the platform server 610,while other systems 620-670 may communicate with the platform server 610indirectly via one or more intermediary network components (e.g.,routers, gateways, firewalls, etc.) or other devices (e.g., contentmanagement servers 102, content servers 112, etc.). Although thedifferent communication networks and physical network components havenot been shown in this example so as not to obscure the other elementsdepicted in the figure, it should be understood that any of the networkhardware components and network architecture designs may be implementedin various embodiments to support communication between the systems,servers, and devices in the digital credential management system 600.Additionally, different systems 620-670 may use different networks andnetworks types to communicate with the platform server 610, includingone or more telecommunications networks, cable networks, satellitenetworks, cellular networks and other wireless networks, andcomputer-based IP networks, and the like. Further, certain componentswithin the digital credential management system 600 may include specialpurpose hardware devices and/or special purpose software, such as thoseincluded in I/O subsystem 611 and memory 614 of the platform server 610,as well as those within the memory of the other systems 620-670, and thedigital credential data store 615 maintained by the platform server 610,discussed below.

Although the various interactions between the platform server 610 andother systems 620-670 may be described below in terms of a client-servermodel, it should be understood that other computing environments andvarious combinations of servers and devices may be used to perform thefunctionality described herein in other embodiments. For instance,although the requests/responses to determine the authorized issuers 630for specific digital credential templates, the generation of digitalcredentials, and the retrieval and presentation of digital credentialtracking and reporting data, may be performed by a centralized web-basedplatform server 610 in collaboration with various client applications atthe other systems 620-670 (e.g., web browser applications or standaloneclient software), in other cases these techniques may be performedentirely by a specialized digital credential platform server 610, orentirely by one or more digital credential tools (e.g., softwareservices) executing on any one of the systems 620-670. In otherexamples, a client-server model may be used as shown in system 600, butdifferent functional components and processing tasks may be allocated tothe client-side or the sever-side in different embodiments.Additionally, the digital credential data store 615 may be implementedas separate servers or storage systems in some cases, and may useindependent hardware and software service components. However, in otherimplementations, some or all of the digital credential data store 615may be incorporated into the platform server 610 (as shown in thisexample) and/or may be incorporated into various other systems 620-670.

In some embodiments, each of the systems 620-670 that collaborate andcommunicate with the platform server 610 may be implemented as clientcomputing systems, such desktop or laptop computers, smartphones, tabletcomputers, and other various types of computing devices, each of whichmay include some or all of the hardware, software, and networkingcomponents discussed above. Specifically, any of client systems 620-670may be implemented using any computing device with sufficient processingcomponents, memory and software components, and I/O system componentsfor interacting with users and supporting the desired set ofcommunications with the platform server 610, as described herein.Accordingly, client systems 620-670 may include the necessary hardwareand software components to establish the network interfaces, securityand authentication capabilities, and capabilities fortransmitting/receiving digital credential templates and digitalcredentials, digital credential data requests/responses to the platformserver 610, etc. Each client system 620-670 may include an I/Osubsystem, network interface controller, a processing unit, and memoryconfigured to operate client software applications. The digitalcredential platform server 610 may be configured to receive and executevarious programmatic and graphical interfaces for generating, managing,and tracking issued digital credentials, in collaboration with thevarious client systems 620-670. Accordingly, each client system 620-670may include an I/O subsystem 611 having hardware and software componentsto support a specific set of output capabilities (e.g., LCD displayscreen characteristics, screen size, color display, video driver,speakers, audio driver, graphics processor and drivers, etc.), and aspecific set of input capabilities (e.g., keyboard, mouse, touchscreen,voice control, cameras, facial recognition, gesture recognition, etc.).Different client systems 620-670 may support different input and outputcapabilities within their I/O subsystems, and thus different types ofuser interactions, and platform server 610 functionality may becompatible or incompatible with certain client systems 620-670. Forexample, certain types of digital credential generation and searchfunctionality may require specific types of processors, graphicscomponents, network components, or I/O components in order to beoptimally designed and constructed using a client system 620-670.

In some embodiments, the digital credential platform server 610 maygenerate and provide software interfaces (e.g., via a web-basedapplication, or using other programmatic or graphical interfacetechniques) used by the various client systems 620-670 to perform thevarious digital credential management functionality described herein. Inresponse to receiving inputs from a client system 620-670 correspondingto digital credentials, templates, credential search requests andcriteria, etc., the platform server 610 may access the underlyingdigital credential data store 615 perform the various functionalitydescribed herein. In other to perform the tasks described herein,platform server 610 may include components such as network interfacecontrollers 612, processing units 613, and memory 614 configured tostore server software, handle authentication and security, and to store,analyze, and manage the digital credentials, templates, and credentialtracking data stored within the digital credential data store 615. Asshown in this example, the digital credential data store 615 may beimplemented as separate dedicated data stores (e.g., databases,file-based storage, etc.) used for storing digital credential templateobjects, issued digital credentials, credential tracking data, andauthorized user/role data. The platform server 610 and data store 615may be implemented as separate software (and/or storage) componentswithin a single computer server 610 in some examples, while in otherexamples may be implemented as separate computer servers/systems havingseparate dedicated processing units, storage devices, and/or networkcomponents.

Certain aspects described herein related to the testing andcertification processes used to verify the skills or qualifications thata user (or earner) has obtained in order to be awarded with a digitalcredential (or badge) or any other skill certification from aninstitution or credentialing body. In some embodiments, physical testingenvironments including “simulation laboratories” may use implemented toallow users to perform physical tasks (including mental and/orcomputer-based tasks) in a monitored environment. Such physical testingenvironments may use virtual reality and/or augmented reality in variouscases. The simulation lab and/or the user may be monitored by varioussensors during testing or certification processes, and the results maybe analyzed to determine (at least in part) whether or not the usershould be awarded a particular digital credential or certification. Asdiscussed below in more detail, simulation labs may be implemented astesting environments for manual tasks, computer-based tasks, scenariotraining, etc., and various monitoring of the simulation lab environmentduring test may provide data metrics relating to successful completionof tasks, efficiency of task completion, user response times, userdecision making behaviors, user biometrics and risk factors, etc.Further, as discussed below, certain simulation labs may provide theability to change testing scenarios as well as environmental conditions(lighting, noise, temperature, etc.) during testing.

Referring now to FIG. 7, an example is shown of a physical testingenvironment that may be used for badge testing, skills certification,and other behavior monitoring and digital credential generation, inaccordance with certain aspects described herein. In this example, abasic testing environment 700 is shown to illustrate certain featuresand concepts that may be included in various embodiments. Depending onthe particular digital credential, activity, skill or ability to beverified, different devices and components may be included in thesimulation environments 700. For example, simulation environments 700for standardized testing and completion of computer-based tasks may besetup to simulate an office environment, for instance, with a computer,keyboard, monitor, desk and chair, etc. Other testing environments 700designed for other badges and/or skills certifications may be configureddifferently. For instance, testing environments 700 may be configured asa driving simulator (e.g., having front and side display screens, aninstalled automobile seat with steering wheel, pedals, vehicle controlsand gauges, simulated mirror displays, etc.), or a flight simulator(e.g., having front and side display screens, up and down fields ofvision, a pilot seat with a center stick and/or other airplane controlsand gauges, etc.). Other testing environments 700 might not require orhave any display screens, for example testing environments 700 for CPRcertification may include one or more CPR manikins and other accessoriesto test CPR scenarios. Additional testing environments 700 may beimplemented for law enforcement use of force or defensive tacticsscenarios (with or with display screens, with or without live firearmscapabilities, etc.). Still other testing environments 700 may beimplemented for skills testing and verification on machine assemblytasks, and/or on machine use tasks. The machines in testing environments700 in such scenarios may range from simple to complex, to allow usersto any testable task on any machine, from bicycle assembly, toautomobile maintenance, to semiconductor design, to electrical work, tolaser fabrication, to welding. Other testing environments 700 may beimplemented for skills testing and verification in performance ofmedical or dental procedures, and the like, and thus may resemble ahospital operating room or dentist office with a full complement ofmedical tools and devices necessary to perform the tasks to be verified.Still other testing environments 700 may be configured to test/verifyskills with respect to sports or other physical activities, and thus thetesting environments 700 may comprise a dance studio, gymnasticsapparatus, golf driving range, or other sports equipment. For each ofthese examples, and many others, it should be understood that thedifferent configuration of testing environments 700 may requiredifferent sets of testing equipment, as well as different monitoring andenvironmental control features. Further, although many examples andimplementations described herein refer to human users as the subjects oftesting and simulation scenarios, in some cases the test subjects mayinclude mechanical devices (e.g., machines configured to assembleparts), artificial intelligences and/or other software programsconfigured to perform certain tasks, etc.

In addition to the testing equipment and apparatuses in the physicaltesting environment 700, the environment may have cameras 705 andsensors configured to monitor the performance and behavior of the userduring the testing. As shown in this example, a number of cameras 705may be installed throughout the testing environment 700 to captureimage/video data of the user from different angles during thetesting/skills verification process. In addition to cameras, in variousembodiments (depending on the type of test or skill being evaluated),additional sensors may be deployed within the testing environment 700,including microphones, light sensors, heat sensors, vibration sensors,and any other sensor type, depending on the type of testing/evaluationbeing performed. For instance, for testing of computer-based tasks,additional sensors such as mouse movement trackers, keystroke loggers,and user eye-tracking software may be used. For machine usage tasks,scenario training, and the like, movement sensors may be placed on theuser and/or on any objects with which the user may interact during thetesting scenario. Additionally, for any testing or skills evaluationscenario, certain embodiments may include biometric sensors and devices710 configured to detect and track the user's biometric data during thetesting process. Such biometric sensors and devices may measure theuser's temperature, heartrate, blood pressure, respiration, skinconductivity, body movements, brainwave activities, etc.

In some embodiments, the physical testing environment 700 also mayinclude various environmental controls that allow a test administratorto control the physical environmental conditions during a test or skillsevaluation. Such environmental controls may include lights 715 thatallow the test administrator to control the light levels, angles, and/orcolors during a test. By way of example, lighting control within theenvironment 700 may allow the test administrator to evaluate the user'sability to perform a driving maneuver or roadside maintenance task atnight, etc. Additional environmental controls may include may includetemperature controls, weather simulation (e.g., wind, rain, snow,sunshine, fog, etc.), speakers to provide background noise ordistraction, olfactory control that provides scents/odors to simulatethe smells that be present during a comparable real-life scenario,vibration control to simulate the activity, and so on.

Referring now to FIG. 8, a flow diagram is shown illustrating an exampleprocess of executing tests or simulations, as well as monitoring andanalyzing the results of the tests or simulations. As described below,the steps in this process may be performed using various components of asimulation lab and/or other physical test (or simulation) environment700, described above. For example, each of steps 801-810 may beperformed by a computer server of a test administrator associated with aphysical simulation environment 700. In other examples, physicalsimulation environments 700 might be configured to receive test contentand configuration parameters, to execute the tests and monitor theexecution, and then to transmit the test results and related observationdata to a separate server (e.g., a digital credential platform server610) for scoring and analysis.

In step 801, a computer server controlling the physical testingenvironment 700 may receive input relating to the test or skillsevaluation scenario to be executed within the physical testingenvironment 700. In step 802, the server may receive data identifyingthe particular user designated to complete the test or skills evaluationscenario.

In step 803, the server may retrieve the test or scenario to beloaded/executed within the physical testing environment 700. As notedabove, the test or scenario may include interactive user software (e.g.,driving or flight simulator programs, law enforcement scenarios, etc.)and/or may include testing software or other software programs loadedonto a desktop, laptop, or tablet computer. For instance, the test orscenario may require the user to work with computer-aided designsoftware, spreadsheet software, database development software, etc. Inother cases, the test or scenario may include audio and/or video filesto be played via speakers and/or display screens within the physicaltesting environment 700, such as instructional videos or audio/visualtest questions.

The test or scenario retrieved in step 803 also may be retrieved basedon the identity of the particular user who will be completing the testor skills evaluation scenario. In some embodiments, the server of thephysical testing environment 700 may be configured to select theappropriate test, scenario, and/or simulation (e.g., a particularsoftware scenario, skill level, etc.) based on the user's current set ofbadges or digital credentials, the user's skill level, and/or the user'sperformance history on previous tests or scenarios within the testingenvironment 700. Additionally, in some cases, the server may varyscenarios/test questions so that a particular user does not receive thesame test questions, scenarios, or other testing content that they havealready completed (or completed within a particular recent time window).

In step 804, the server may determine and apply a set of environmentalconditions within the physical testing environment 700 for the executionof the test or scenario. As noted above, the physical testingenvironment 700 in some embodiments may be capable of setting variousenvironment conditions such as lighting (e.g., to simulate different dayor night, and/or different real-world working environments), temperatureand weather conditions (e.g., to simulate outdoor scenarios, differentseasons and locations), noise (e.g., to provide background noise,traffic noise, distractions, etc.) and other various environmentconditions. The server may select and apply environmental conditions aspart of the test or scenario selected in step 803, or as a separatedetermination which is performed based on random chance or selected by atest administrator, etc. For instance, for certain types of badges andother certifications, separate day and night testing of certain tasksmay be required. In other cases, the environmental conditions may beselected randomly and changed for each testing session. In still othercases, user may select and/or save their preferred environmentalconditions for different types of testing. Further, in some embodiments,the physical testing environment 700 may track and analyze the user'svarious testing or scenario performance metrics (e.g., accuracy,efficiency, safety, compliance, biometrics, etc.) under differentenvironmental conditions, in order to determine the optimalenvironmental conditions for the particular user. In such cases, user'smay receive different badges or certifications (or may have differentbadge assigned characteristics or endorsements) based on their test orscenario performance in different environmental conditions.

In step 805, the computer server(s) associated with the physical testingenvironment 700 may execute the test or simulation scenario, duringwhich the user's performance and any/all user reactions or responses maybe monitored. As noted above, even for certain tests that are entirelymanual in nature, the physical testing environment 700 may use camerasand any other sensors to monitor the user's actions. Such monitoring mayinclude various aspects of the user's performance, such as answers totest questions selected via a testing computer terminal, or the user'sinteractions with physical objects (and/or other people) within thephysical testing environment 700. The user's answers and actions may berecorded by cameras and computer input devices, and additional user datamay be collected using various other sensors such as microphones,biometric sensors, etc.

In step 806, the results for the test and/or simulation scenariocompleted by the user may be analyzed. In some embodiments, the suchanalyses may be performed based not only on the user's responses toparticular test questions or scenarios. Additionally or alternatively,the analysis in step 806 may include an evaluation of the user's otherreactions or responses, such as speed and confidence of action (e.g., asdetermined by user comments, speed of response, facial expressionanalysis, body movement analysis, biometric data, etc.), efficiency,safety, decision making, and user biometrics. One or more of theseseparate analyses may be performed in steps 807-810, and each may beperformed independently of the others, or may be combined into a singleanalysis. For instance, in some cases the goal of the simulation mightbe only to measure the user's biometric data, and the user's actualresponses to the questions/scenarios may be irrelevant and need not beevaluated in step 807. In other tests or simulation scenarios, theopposite analysis may be applied, where only the accuracy of the user'sresponses or behaviors are measured and analyzed in step 807, and theuser's biometric data is irrelevant and thus the analysis in step 810 isnot performed. As another example, in a certain simulation of driving,machine operation, use of force training, etc., the only relevantanalysis to be performed may be a safety/decision making analysis instep 809, while the efficiency analysis in step 808 need not beperformed. In other similar tests/situations, the server may apply botha safety/decision making analysis in step 809 and an efficiency analysisin step 808 (e.g., to confirm that a driving maneuver or route wascompleted both safely and efficiently, to confirm that a suspect wassubdued safely and quickly, to assure that a manufacturing assembly taskwas performed safely and efficiently, etc.).

For example, in certain embodiments and implementations of the conceptsdiscussed above in reference to FIGS. 8-9, various techniques (e.g.,systems, methods, computer-program products tangibly embodied in anon-transitory machine-readable storage medium, etc.) may includeevaluating a physical simulation by a digital credential generatorsystem (e.g., 630 and/or 610). Such techniques may include monitoring,by a digital credential generator system, a physical simulation areausing a plurality of sensors, during a physical simulation. During themonitoring, the digital credential generator system may detect, usingthe plurality of sensors, various physical actions performed by a userduring the physical simulation. The digital credential generator systemmay analyze data corresponding to the plurality of physical actionsperformed by the user during the physical simulation, and determine thatthe user corresponds to a particular credential receiver. The digitalcredential generator system then may determine whether the firstcredential receiver is eligible to receive a digital credential, bycomparing the data corresponding to the analysis of the physical actionsperformed by the user during the physical simulation, to one or moredigital credential requirements. Finally, the digital credentialgenerator system may generate a first digital credential associated withthe first credential receiver and with the digital credentialrequirements, in response to determining that the first credentialreceiver is eligible to receive the first digital credential.

In some embodiments, outputting a physical simulation may includeoutputting audio and/or video simulation components within the physicalsimulation area, manipulating physical objects (e.g., motorized objects)during a live-action simulation within the physical simulation area,and/or outputting virtual reality simulations via a virtual realityheadset. Additionally, certain embodiments may include generatingphysical simulation environments within a physical simulation area,including, for example, simulating ambient light conditions within thephysical simulation area, outputting one or more background noiseconditions within the physical simulation area, monitoring andcontrolling the physical temperature using a heating and cooling systeminstalled at the physical simulation area, outputting smells to thephysical simulation area using a smell output device, and/or outputtingvibratory effect within the physical simulation area, using a vibrationsystem.

As noted above, the monitoring of a test/simulation may includemonitoring physical actions/activities performed by the user using videorecording devices and/or motions. Additionally or alternatively, themonitoring may be of computer-based tasks, using additionalsoftware-based sensors such as mouse movement trackers, keystrokeloggers, and user eye-tracking software, etc.

In accordance with certain aspects described herein, the processes usedfor testing/evaluating a user and determining that a user is eligiblefor a particular digital credential (or badge) need not include aspecific test, designated evaluation, or scored scenario training.Rather, the testing and badging determinations may be performedautomatically during the user's normal course of on-the-job performanceof tasks. In such embodiments, the testing and credentialing of usersmay be based on observation of workers during their normal workactivities. Cameras and other sensors may be installed and used todetect the completion of tasks and/or certain competencies of the users,and the data from these sensors may be evaluated to automaticallydetermine when the user is eligible for a digital credential. Thus, onthe job testing and badging may be performed entirely transparently tothe worker performance of their job duties, and need not require anydelay or distraction from job performance, or any designated time orlocation needed to perform formal testing.

In order to perform automatic and on-the-job testing and credentialingof workers or other users (e.g., students, athletes, etc.), the “work”environment of the user may be monitored with cameras and/or sensorscapable of tracking the user's activities and performance. As discussedabove with respect to the implementation of physical testingenvironments (e.g., 700), different types of digital credentials relateto different activities that may be performed in a variety of differentwork environments. Referring briefly to FIG. 9A, an example workenvironment 900 a is shown for a user completing computer-based tasks.In this example, the work environment 900 a may include a basicworkstation, server, modem, printer, monitor, keyboard, etc., as well asdesk and chair to allow the user to complete normal computer-based workactivities. In this example, the user may be data entry specialist,computer programmer or design engineer, call center customer supportoperator, or may be performing any other computer-based job. In suchexamples, sensors 905 and 910 may include cameras, network monitoringdevices, keystroke loggers, mouse movement monitors, biometric devicesand sensors, etc. Such software tools may operate as backgroundprocesses on a computer terminal being monitored. Additional monitoringdevices may be built into specific software programs with which the useris interacting, and may be able to determine the correctness, quality,and efficiency of the user's interaction with the particular software.For example, if a user is interacting with a spreadsheet softwareapplication or computer-aided design application to perform a work task,then monitoring features within the software application may be used todetermine how quickly the user performed the task, how many attempts ittook the user, how correct/accurate was the finished product, etc. Inother examples, the monitoring of the user's interaction with aparticular software program need not involve any monitoring featureswithin the software itself, but instead may include monitoring at theoperating system or hardware layers, or monitoring that is entirelyexternal to the workstation. For example, external cameras 905 and othersensors may capture and analyze the user's interactions with thesoftware application, and thus need not affect the operation of thesoftware at all.

Another example work environment is shown in FIG. 9B. In this examplework environment 900 b, the entire layout of workplace floor is shownand monitored by a series of cameras 905 and/or other sensors. Themonitoring in this example may apply to works who do not perform onlycomputer-based tasks, but whose work requires them to interact withphysical objects within their workspace, and/or to move around the workenvironment 900 b to other workspaces. For instance, maintenance works,office mail delivery works, construction workers, electricians,plumbers, machine assembly or manufacturing works, etc., may bemonitored with such systems. When monitoring a larger area for theperformance of non-computer-based work tasks, in addition to cameras905, the work environment 900 b may include motion sensors, microphonesand noise sensors, as wells as movement sensors and/or tracking devicesthat may be placed on specific physical objects within the environment.By way of example, work environment 900 b may correspond to a shopfloor, mechanic's garage, or manufacturing assembly plant, and thecameras 905 and other sensors may be used to confirm that workers arecomplying with safety requirements and/or health codes with respect withtheir work with machinery or hazardous materials, etc. As anotherexample, work environment 900 b may be an office environment, and thecameras 905 and other sensors may be used to confirm that individualworkers are working efficiently, in their assigned areas, etc., and thatworkers without assigned areas (e.g., cleaning, mail delivery,maintenance workers, etc.) are working efficiently and not skipping anyportion of the floor 900 b.

Referring now to FIG. 10, a flow diagram is shown illustrating anexample process of automatically monitoring work activities and issuingdigital credentials via “on-the-job” testing. As described below, thesteps in this process may be performed by monitoring and credentialingcomputing devices operating within various types of work environments900, such as those described above. For example, each of steps 1001-1006may be performed by a computer server operating automatically andunassisted (or at the direction of an administrator) within a workenvironment 900. In other examples, work environments 900 might beconfigured only to monitor work activities and performance, and then totransmit the results and related observation data regarding variousworker to a separate server (e.g., a digital credential platform server610) for scoring, analysis, and the issuance of digital credentials.

In step 1001, a computer server controlling the on-the-job badgingsystem may activate the cameras, sensors, monitoring software, etc.,within the workstation and/or work environment. As discussed above, thisactivation may include specific monitoring software to detectcomputer-based tasks, and/or location monitoring devices such ascameras, sensors, biometrics, etc., depending on the type of workers andwork environments 900 being monitored. In some cases, an on-the-jobtesting and credentialing system may be implemented as an “always on”system, in which the workstation/workplace monitoring is constantlyrecording and analyzing worker activities. Thus, step 1001 may beoptional in such embodiments. However, in other cases,workstation/workplace monitoring might only be activated at certaintimes and not others, for example, only during normal work hours, onlyon certain specific work days designated for work evaluation, etc. Insome embodiments, a system administrator and/or individual workers mayactivate or de-activate the workstation/workplace monitoring systemswithin their work environment at any time. Thus, such systems need notbe an invasion of privacy for any worker that does not choose for theirwork to be monitored and evaluated, but workers may choose to turn themonitoring systems on in order to be eligible for evaluation and earningof additional work related digital credentials and credentials.

In step 1002, the workstation/workplace monitoring systems may capturethe user's work-related activities and behaviors, including performingvarious computer-based tasks and non-computer-based tasks as discussedabove. In step 1003, the user's working data as collected by theworkstation/workplace monitoring systems and sensors may be analyzed bythe server, in order to determine in step 1004 whether or not the useris eligible for one or more digital credentials or other credentials(e.g., professional certifications, etc.) based on their on-the job workactivities. Certain digital credentials or credentials may be madeavailable to users in response to detecting that the user has successfulcompleted one or more specialized work tasks, thus demonstrating thatthe user has obtained the particular skill associated with the digitalcredential. In some cases, the server and/or the monitoring systems andsensors may also be configured to detect a certain level of efficiencyby the user in performing the tasks, and/or may require that the userperform a certain task N number of times before the user is eligible forthe digital credential or credential.

In step 1004, if the system determines that the user is eligible for oneor more particular digital credentials (1004:Yes), then in step 1005 thesystem may either issue the digital credential directly (e.g., if theworkplace server is permitted to be digital credential issuer), and/ormay initiate a communication session with a badging platform 610 and/ordigital credential issuer 630 to request that a new digital credentialis issued for the worker. In such examples, the workplace server mayprovide the information identifying the worker (e.g., name, employee ID,digital credential system profile ID, etc.) to a digital credentialplatform 610 or issuer 630, along with verification that the worker hascompleted the requirements to earn a particular digital credential. Insome embodiments, the servers operating at the workplace may beconfigured to capture evidence (e.g., video evidence, screen captures,facial/identity verification, etc.) and transmit the evidence to thedigital credential-issuing authority, before the digital credential maybe issued.

In step 1006, the worker may be notified that they have received adigital credential based on their normal on-the-job activities. In someembodiments, the worker may indicate interest in obtaining one or moreparticular digital credentials, and the workstation/workplace monitoringsystem may be configured to evaluate the worker with respect to theparticular digital credentials or credentials that the worker hasexpressed interest in. However, in other examples, it may be possiblefor a worker to receive an issued digital digital credential withoutexpressing any interest in the digital credential (or even being awareof such a digital credential), but solely based on the determinationthat the worker has achieved the level of skills mastery required forthe digital credential/credential, based on the automated monitoring ofthe worker within the workplace. In certain cases, a user may beinformed that they are eligible for receiving a digital credential priorto the issuance of the digital credential in step 1005, and the user maybe allowed to accept or reject the digital credential. Additionally, insome cases, the user may receive status reports (e.g., daily, weekly,etc.) identifying which digital credentials the user is being monitoredfor, and the user's progress with respect to earning those digitalcredentials. This data may include indications to the worker that he/shemay earn a particular digital credential after performing a task anotherN times, or performing the task N amount faster, or performing the taskwithout making any errors or backtracking, etc.

For example, in certain embodiments and implementations of the conceptsdiscussed above in reference to FIGS. 10-11, various techniques (e.g.,systems, methods, computer-program products tangibly embodied in anon-transitory machine-readable storage medium, etc.) may includegenerating digital credentials for particular credential receivers,based on monitoring of a physical environment using a plurality ofsensors. A digital credential issuer (or generator) 630 and/or 610 maydetect or receiver sensor data from a number of sensors corresponding tothe user actions performed by a user within the physical environment.The digital credential generator may determine the operations thatwere/were not performed within the physical environment, based on theuser actions detected. The digital credential issuer then may retrievedata from a credential receiver data store associated with theuser/credential receiver, and determine one or more digital credentialtemplates, based on the retrieved data associated with the firstcredential receiver. For each of the digital credential templates, thedigital credential issuer/generator may retrieve criteria associatedwith the digital credential template, compare the operations performedby the user within the physical environment to the criteria associatedwith the digital credential template, and then determine whether or nota credential receiver is eligible to receive a digital credential basedon the digital credential template. If the first credential receiver iseligible to receive a digital credential based on the digital credentialtemplate, the digital credential generator may generate a digitalcredential based on the digital credential template and user dataassociated with the first credential receiver.

In such cases, the generated (or issued digital credential) may beembedded with additional data such as the evaluation/simulation time,location, or the sensor system/physical environment within which theevaluation/simulation was performed. Additionally, in some cases, facialrecognition data and/or biometric data may be collected from the user(credential receiver), and may be used to validate or authenticate thedigital credential by verifying the user's identity. As in the aboveexamples, the monitoring may be done using physical movement trackingsensors such as video recorders and/or motion detectors, or may usesoftware-based sensors such as network monitoring devices, keystrokeloggers, mouse movement monitors, touch screen monitors. Suchsoftware-based tools may operate as background processes on a computerterminal being monitored, and/or may be built into specific softwareprograms with which the user is interacting.

Additional aspects related to the automated tracking of user or workeractivities, after the user/worker has been issued a badge (or digitalcredential), in order to determine how often the user/worker is “using”their digital credential. Depending on type of digital credential orcredential, post-credentialing monitoring of the user may involveanalysis of user's physical work product (e.g., documents produced,parts/items created, etc.), or may be involve observations of the user(e.g., via a workstation/workplace monitoring system). In order toevaluate how often a user is using a particular digital credential, adata store of digital credentials may be linked to particular skills,work-related, or activities. The user/worker may then be tracked todetermine the number of such tasks performed, and/or the quality,efficiency, and/or competence of the user's performing those tasks, inorder to determine to what extend the user/worker is “using” the digitalcredential.

Referring now to FIG. 11, an example computing environment 1100 isshown, including a digital credential platform server 1110, one or moreworkstation/workplace monitoring systems 1120, and acredential-to-activity mapping data store 1130. In some examples, thedigital credential platform server 1110 may be a badging server similaror identical to the server 610 discussed above. Thus, server 1110 may beconfigured as a digital credential repository and credentialing system,acting as a clearinghouse for digital credential owners, issuers,earners, endorsers, etc. Server 1110 may include a digital credential(or digital credential) data store configured to store badginginformation such as the details of the particular digital credentialsearned by particular users. As noted above, such details may include thedate on which a digital credential was issued to a user, and for certaindigital credentials, an expiration date associated with the digitalcredential.

In this example, system 1100 also includes a credential-to-activitymapping data store 1120, which may be implemented as a separate externaldata store and/or may be integrated into the digital credential datastore of server 1100. The credential-to-activity mapping data store 1130may include mappings of one or more tasks or activities associated witheach digital credential type that a user may potentially earn. Forexample, a digital credential relating to automotive maintenance for aparticular make of car may have associated activities and tasks thatinclude particular maintenance tasks (e.g., tune-ups, part replacements,etc.) for different model cars having the make. As another example, anoperating system administrator-related digital credential may list,within data store 1130, various system administrator tasks and that auser may perform on the particular operating system. In some cases, theactivities or tasks associated with a particular digital credential maycorrespond to the same set of activities or tasks that a user isrequired to perform to earn the particular digital credential, and asdiscussed below, these activities or tasks may serve as a metric toevaluate how much the user is “using” the digital credential.

One or more workstation and/or workplace monitoring systems 1120 mayprovide user monitoring data to the server 1110, to allow the sever 1110to analyze the user's activities and determine to what extent the useris using the activities and abilities associated with their digitalcredentials. In some embodiments, the workstation and/or workplacemonitoring systems 1120 may be similar or identical to any of theworkstation/workplace monitoring systems and sensors discussed above.For example, workplace monitoring systems 1120 may collect recordsdetailing the user's physical work product (e.g., documents produced,modified or accessed by the user, inventory or work order recordsindicating tasks performed by the user, etc.). Additionally, workplacemonitoring systems 1120 may include observation systems (e.g., workplacemonitoring systems) including cameras and other sensors to track theuser's activities and determine which specific tasks have been performedby the user.

In some embodiments, the monitoring and tracking of post-credentialingactivities by the user may be used to analyze and provide digitalcredential or credential feedback data to various entities. For example,referring now to FIG. 12, a flow diagram is shown illustrating anexample process that may be used to determine whether a user has or hasnot used the activities associated with a particular digital credentialthat they have obtained, and then to aggregate and report that digitalcredential usage data to the relevant parties. In step 1201, aparticular digital credential is issued to a user based on the user'ssuccessful completion of the badging requirements. As in the variousexamples discussed above, the digital credential may be associated witha computer-based activity, non-computer-based activity, or any other setof digital credential requirements determined by a digital credentialowner or issuer. Additionally, the digital credential issuance in step1201 may be the result of formal testing and/or certification processes,or may be based on on-the-job or other observational data collected forthe user.

In step 1202, the digital credential server 1110 and/or monitoringsystems 1120 may monitor and track the activities of the credentialeduser, including, for example, the workplace tasks performed by the userbased on analyses of the various monitoring systems/sensor datainstalled at the user's workstation and/or workplace environment. Asdescribed above, determining what activities and tasks the credentialeduser has performed, and when, may be performed using a variety oftechniques. In some cases, determining what work-related tasks a userhas performed, and what other activities they have been engaged in, maybe done by analyses of written and electronic documents associated withthe user or workplace. For instance, documents such as maintenancerequests, work orders, customer tickets, purchase receipts, and the likemay be analyzed to determine what activities or tasks the user hascompleted and when. For instance, a maintenance record listing the useras the assigned technician may be used in determination that the userhas performed the specified task/activity at the time listed on therecord. In other examples, the user's electronic mail and otherelectronic documents may be searched and analyzed (e.g., using a keywordanalysis and/or trained artificial intelligence) to determine what tasksthe user has performed and/or what activities the user has demonstratedduring the relevant time periods. In some embodiments, there may beparticular advantages in implementing a post-credentialing usageanalysis and/or digital credential valuation process for certain digitalcredentials/tasks that are more discrete and detectable, for instance, anumber of transmissions changed after earning a vehicle transmissioncertification, a number of particular medical procedures done followinga digital credential credential for the procedure, a number of ITtickets resolved successfully following receiving an advanced ITcomputer services and computer repair digital credential, etc. Incontrast, for other tasks and activities for which a user may receive adigital credential, such as leadership, communication skills, advanced Csoftware programming, jujitsu skill levels, and the like, it may be moredifficult to quantify if, when, and how often a user is using theparticular skill or task associated with the digital credential.

In step 1203, a set of tasks and/or activities associated with thedigital credentials obtained by the specific user may be retrieved usingthe credential-activity mapping data store 1130, and in step 1204 theretrieved tasks and/or activities may be compared to the tasks andactivities that have been performed by the user subsequent to thedigital credentials being earned (as determined in step 1202). As anexample, the comparison in step 1204 may determine that in the six monthsince the user was issued a professional certification to perform aparticular technical task, the user has performed that task on a weeklybasis. Alternatively, for a different digital credential issued to theuser directed to expertise in a particular software program, thecomparison in step 1204 may determine that the user has used thatsoftware program only once since receiving the digital credential twoyears ago. In this case, the system may conclude that the professionalcertification issued six months ago to the user has been of greaterusefulness than the software digital credential issued two years ago(allowing for the possibility of career changes, prestige-driven digitalcredentials rather than functional digital credentials, etc.).

In step 1205, data from the comparison of step 1204, i.e., dataindicating the post-credentialing usage by the user of the digitalcredential-associated activities or tasks, may be aggregated andanalyzed, and then transmitted to one or more of the relevant systemcomponents. In various embodiments, any of several different componentsand roles associated with the credentialing platform 1110 may requestand receive this information for their associated digital credentialsand/or associated users. For instance, digital credential owners and/ordigital credential issuers may request and receive from the platformserver 1110 data regarding the post-issuance usage of the digitalcredentials they own or have issued. In other cases, digital credentialendorsers may request and receive from the platform server 1110 dataregarding the post-issuance usage of the digital credentials they haveendorsed. Digital credential earners, the users themselves also mayrequest reports from the platform server 1110 quantifying thepost-credentialing usage (which may be expressed in terms of time,value, and/or dollar amounts) associated with their previously earneddigital credentials. Employers and other organizations also may requestsuch reports for their employees or organization members, in order todetermine which digital credentials have been the most used and mostuseful to the organization.

Referring now to FIG. 13, another flow diagram is shown illustrating anrelated process involving determining whether a user has or has not usedthe activities associated with a particular digital credential that theyhave obtained, and then adjusting an expiration or re-certification dateassociated with the digital credential based on the user's usage of thedigital credential skills. The steps in this example may be similar oridentical to the corresponding steps in FIG. 12, and in someembodiments, the analyses and transmission of the post-credentialingusage described in step 1205 may be performed in conjunction with thesetting of an expiration or re-certification date for the digitalcredential as discussed below.

Steps 1301-1304 may correspond to steps 1201-1204 in some cases, and maybe performed using similar or identical techniques to those discussedabove. For example, in step 1301 a platform server 1110 and/or digitalcredential issuer may issue a digital credential associated with one ormore activities or tasks to a particular user, recording the digitalcredential issuance data within the digital credential data store. Instep 1302, the post-issuance activities of the particular user may bemonitored, including monitoring of the user's work-related activitiesand tasks performed/completed, in order to determine the particulartasks and activities with which the user has been engaged followingissuance of the digital credential. In step 1303, the skills,activities, and tasks associated with the user's digital credential(s)are retrieved, and in step 1304 are compared to the post-issuance usertasks and activities determined for the user in step 1302. Finally, instep 1305, based on the comparison in step 1304, the platform server1110 may determine that an expiration date and/or recertification dateassociated with the user's digital credential should be adjusted basedon the user's post-issuance activities. As an example, if the systemdetermines in step 1305 that a user who received a digital credentialcorresponding to a forklift operator's license or commercial truckdriving license three years ago, but has infrequently (or not at all)driven a forklift or a commercial truck since receiving their digitalcredential, then the system may determine that the user's license shouldexpire at the earliest possible time (e.g., the expiration time as ofwhen the digital credential was first issued). In contrast, if thesystem determines in step 1305 that the same user has frequently andconsistently driven a forklift or a commercial truck ever sincereceiving their digital credential, and also that the user has ahigh-safety rating and/or high safety compliance scores, then the systemmay determine that the user's license may be extended. In such cases,the platform server 1110 may determine a new extended expiration orrecertification time for the digital credential, update the user'sdigital credential record within the digital credential data store, andtransmit notifications to the affected entities (e.g., the user,employer, digital credential issuer, digital credential owner, etc.)providing the new expiration date. In other examples, rather thanchanging the expiration date or recertification date of a digitalcredential (or eliminating the expiration altogether), the platformserver 1110 may in other examples determine a new recertification courseor procedure for the user, such as simple refresher course to allow theuser to recertify quick than the longer complete recertification courseused by other users with less post-credentialing digital credentialusage.

Additional aspects described herein relate to capturing and using“evidence” data in connection with user testing and credentialingsystems, on-the-job evaluation and badging systems, and/orpost-credential monitoring systems. For example, within any automatedbadging/certification/verification system, evidence of the user'sperformance may be extracted and saved, for example, in a digitalcredential server along with an associated issued digital credential, oras part of a separate user portfolio of evidence. Evidence data mayinclude, for example, audio and video of the user during a livesimulation, or during a virtual reality or augment reality simulation,audio and keystroke data from the user during the testing processing,the user's reaction time and/or decision-making data during asplit-second simulated scenario or relevant real-life event (e.g., aworkplace accident, etc.), and/or any other sensor or biometric datacollected during testing, credentialing, and/or monitoring. As discussedbelow, evidence data associated with a user may be saved with the user'sdigital credential and/or into a separate portfolio of evidence, whichmay be available to the user for review, and also may be provided uponrequest to potential employers for review during a review or hiringprocess. Such evidence data also may be applied to updated digitalcredential credentialing requirements, so that in some cases a user maysimply resubmit their evidence portfolio instead of being required torecertify their digital credential when the test or credentialingstandards are updated.

Referring now to FIG. 14, an example computing environment is shownincluding a digital credential platform server 1410 in communicationwith a plurality of testing, credentialing, and/or monitoring systems1421-1423, and one or more external client devices 1460. In someexamples, the digital credential platform server 1410 may be a digitalcredentialing server similar or identical to the server 610 discussedabove. Thus, server 1410 may be configured as a digital credentialrepository and credentialing system, acting as a clearinghouse fordigital credential owners, issuers, earners, endorsers, etc. Server 1410may include a digital credential (or digital credential) data storeconfigured to store digital credential information such as the detailsof the particular digital credentials earned by particular users. Asnoted above, such details may identify the digital credential issuerand/or other testing/credential authorities responsible foradministering testing or simulation scenarios as part of the digitalcredentialing process, and/or for pre-digital credential or post-digitalcredential monitoring of workstations/workplaces to detect and analyzeuser tasks performance and user skills/abilities

In this example, the platform server 1410 may receive data from threetesting/credentialing systems 1421-1423. Similar to the above examples,the simulation lab system 1421 may correspond to a simulation lab orother physical testing environment, an on-the-job credentialing systems1422 may include workstation/workplace monitoring systems and sensors torecord and analyze the user's on-the-job performance, and may issuedigital credentials in some cases without the need for any separateformal testing procedure; and post-credential monitoring systems 1423may be configured to monitor users following the issuance of a digitalcredential, including tracking task performance data, skills usage, andthe like, and comparing the data to the skills/tasks associated with theuser's digital credentials.

In some embodiments, one or more systems 1421-1423 which perform usertesting, credentialing, and/or monitoring, such as those systemsdiscussed above, may capture and transmit “evidence data” of the userduring a test, simulation, or during an on-the-job monitoring process.Evidence data may include, for example, video and/or audio of the userduring a test, simulation (e.g., live, VR, or AR), collected by thesensors of a physical testing environment 700. Additional evidence datamay include user reaction time data, decision-making data, facialexpression and body language data, keystroke and mouse movement data,and/or user biometric data. The evidence data may correspond to a timeperiod just before, during, and just after a test, simulation, or a taskor activity performed during on-the-job monitoring.

As shown in this example, the various evidence data collected by systems1421-1423 may be transmitted to the platform server 1410 and stored inan evidence portfolio data store. The evidence data collected by thetesting, credentialing, and/or user monitoring systems may be associatedwith a particular user (or users) and with a particular digitalcredential (or digital credentials) that the user is in the process ofearning or using (e.g., for post-credentialing monitoring). Thus, theevidence data may provide documented proof that the user actuallycompleted the digital credential requirements, along with additionalcontextual evidence showing how the user performed during the testing,simulation, or monitoring.

Referring now to FIG. 15, a flow diagram is shown illustrating anexample process by which a testing system, simulator, credentialingsystems, workstation/workplace monitoring system, and the like, maycollect and preserve evidence data related to a user and a digitalcredential. In step 1501, a testing, credentialing, and/or monitoringsystem such as those described above may execute a test, simulation, oruser monitoring process for a particular user in connection with adigital credential that the user is seeking or has already obtained. Theparticular types of tests may include, for example, live simulationsand/or virtual or augmented reality simulations executed within aphysical testing environment 700. In other examples, the testing in step1501 may correspond to an on-the-job credentialing system that monitorsand evaluates a user's workplace tasks and activities, or to apost-credentialing user monitoring system configured to determinewhether the user is using their previously issued digital credentials.In step 1502, during any of these testing, simulation, or monitoringprocesses, the system 1421-1423 may capture evidence data relating tothe user. As noted above, evidence data may include audio or video ofthe user, user reaction time data, decision-making data, facialexpression data, body language data, the user's keystrokes and mousemovement data, particular software interaction data, and/or the user'sbiometric data. In step 1503, the evidence data may be encapsulated andtransmitted to the platform server 1410 for storage within the user'sevidence portfolio, and in step 1504 the platform server 1410 may storethe evidence data files with data records associated with the user andthe particular digital credential(s) to which the evidence applies. Inother embodiments, certain systems 1421-1423 may retain and store userevidence data locally, rather than the evidence data being stored in acentral repository. Additionally, when the evidence data is transmitted,it may be compressed and edited as needed, and/or encrypted in order toassure data security and user privacy.

In some cases, the platform server 1410 may determine a subset of theuser activities matching digital credential requirements associated withthe digital credential, wherein other user activities might bare norelevance to the requirements of the digital credential. In such cases,the platform server may store only the corresponding subset of theevidence/sensor data for the user activities matching digital credentialrequirements, and might not store other evidence/sensor datacorresponding to irrelevant user activities upon which digitalcredentials do not depend.

Referring now to FIGS. 16A and 16B, two additional flow diagrams areshown illustrating example processes by which evidence data may beretrieved and/or accessed from a platform server 1410 or other datarepository. As noted above, individual evidence data files stored by theplatform server 1410 may be associated with a particular user and/orwith a particular digital credential or credential earned (or in processof earning) by the user. Thus, in some embodiments, evidence data may bestored and made available to certain authorized entities. For instance,in step 1601 of FIG. 16A, the platform server 1410 may receive a requestfor some or all of the user's evidence portfolio. In step 1602, theplatform server 1410 may perform authorization/authentication on therequest to determine (1) whether the requestor is authorized to accessthe user's evidence data, and/or (2) whether the requested evidence iscurrent and valid. One or both of these determinations may requireexplicit authorization from the user himself or herself, in order to (1)prevent any unwanted parties from accessing the user's evidence data,and (2) to prevent any old and obsolete from being accessed, even byauthorized parties. Thus, step 1602 may include verifying therequestor's identity or role and comparing to an access control list orother permissions data associated with the evidence. In some cases, step1602 may include a real-time request sent by the platform server 1410 toa client device associated with the user, to allow the user the optionto allow or reject the request. Additionally, the request in step 1601may specify one or more particular users and/or one or more particulardigital credentials for which the associated evidence is to beretrieved, and thus authorization in step 1602 may be granted or deniedfor evidence relating to each possible combination of users and digitalcredentials. In step 1603, assuming that the requestor has been grantedaccess to the requested evidence data, the corresponding evidence datafiles may be retrieved and forwarded to the requestor.

In some examples, the request in step 1601 may be from the userhimself/herself, who wants to review and study the evidence from his/herprevious tests, simulations, and monitoring data. In other examples, therequest in step 1601 may be from a current or potential employer, whohas been authorized by the user to retrieve and view the user's evidencedata associated with all work-relevant digital credentials, as part of ahiring process or review process. The user's evidence data may verify tothe employer or potential employer that the user actually completed thedigital credential requirements, and also may allow the employer orpotential employer to observe the user's behaviors, responses, reactionsfirst-hand, thus allowing them to evaluate the user's reaction time,efficiency, mental state, decision-making, etc., and other difficult toquantify characteristics. In still other examples, the user mayauthorize a digital credential issuer or digital credential owner toview the user's evidence files related to the digital credentials issuedand owned by those entities. Finally, users may make some or all oftheir evidence data publicly available (e.g., on a file-by-file basis)and/or may actively post their evidence data as a multimedia file ordata records within a digital credential profile page of the user thatis maintained and published by the platform server 1410.

In some embodiments, in addition to (or instead of) providing evidencedata in response to requests, the platform server 1410 may provide thefunctionality to receive updated tests, digital credential requirements,credentialing data, etc., and to apply a user's previously storedevidence to the new testing or credentialing requirements. For instance,in step 1604 of FIG. 16B, the platform server 1410 may receive a requestto apply previously stored evidence data within a user's portfolio to anupdated testing/credentialing process. For example, testing orcredentialing authorities (e.g., a digital credential owners or issuers,employers, etc.) may periodically update digital credentialingrequirements in order to improve the quality of the digital credentialtesting, to comply with new best industry practices, to make a digitalcredential more restrictive by increasing the required scores orefficiency, etc. Additionally, certain testing or credentialingauthorities may implement multiple different levels of the same digitalcredential, in which users are subjected to the same test, samesimulation, same monitoring processes, etc., but different scoringranges may equate to different levels of the digital credential that maybe earned by the user. In these scenarios, whenever digital credentialrequirements are updated, or if a new digital credential level is madeavailable, it may be possible to apply the user's previously collectedevidence data to the new digital credential requirements or digitalcredential level, rather than requiring the user to retake the test,simulation, or monitoring process. As an example, a set of newrequirements for particular digital credential may be similar to theprevious set of requirement, with the addition of a newly imposed timelimit by which the test or simulated scenario must be completed. Inother example, new digital credential requirements or digital credentiallevels may raise the minimum performance level during a test orsimulation to a higher level, and/or may require additional steps orprocedures during the test or simulation that were not required in theprevious version of the digital credential requirements. In these cases,rather than require the user to retest/recertify to earn the updateddigital credential, the platform server 1410 may provide the service ofreceiving the updated digital credential requirements or new digitalcredential levels, and automatically evaluating the new digitalcredential requirements/levels using the user's evidence data that wascollected with earning the previous version of the digital credential.Thus, in step 1605, the requestor may be authenticated and the requesteddata may be validated, and in step 1605 the user's evidence data may beapplied the updated testing/credentialing process. Referring to thesesame digital credential requirements changes discussed above, theevaluation in step 1606 may include automated analysis of the user'sevidence data to determine whether the user complied with the newlyimposed time limit, the new minimum performance level, and/or performedthe additional new steps or procedures during the user's previousdigital credential testing. If so, the digital credential authority mayallow the user to upgrade their digital credential automatically withouthaving to retake the test or simulation, etc. If not, the user may beinformed that they are required to retake the test or simulation (or insome cases they may receive a lower digital credential level). Eitherway, in step 1607, the results of the evidence analysis and applicationto the new credentialing requirements may be output to the requestor.Another potential advantage in certain embodiments may include theprotection of the user's evidence data itself. For instance, in theabove example, the platform server 1410 might perform the analysis andapplication of the user's previously stored evidence data to the newtesting requirement, without ever allowing any other entity access tothe evidence data. In other examples, the platform server 1410 mayperform the analysis and/or may provide the actual evidence data filesto the requestor device, with the sufficient authorization from theuser.

In various embodiments, the updated testing/credentialing process instep 1604 may correspond to a re-issuance of a digital credential, withthe same or updated requirements, or may corresponding to a differentdigital credential having similar and/or overlapping issuancerequirements. For instance, the platform server 1410 may receive anupdated set of requirements for a digital credential previously issuedto a credential receiver, may retrieve the stored set of sensor datacorresponding to the relevant activities performed by the credentialreceiver in connection with the issuance, may compare the retrieved setof sensor data/activities to the updated set of digital credentialrequirements, and then may generate/issue an updated digital credentialto the credential receiver, based on the comparison of the retrieved setof sensor/activity data to the updated set of digital credentialrequirements. Similar techniques may be performed to generate and issuedigital credentials to receivers for entirely different digitalcredentials, rather than updated credentials, such as similarcredentials and/or credentials having overlapping eligibilityrequirements.

Additional aspects described herein relate to capturing and using userbiometric data, physical user cues, and the like, in connection withuser testing and credentialing systems, on-the-job evaluation andbadging systems, and/or post-credential monitoring systems. For example,within any automated badging/certification/verification system, dataidentifying particular physical user cues and/or user biometric data maybe collected during testing/simulation/monitoring processes and saved,for example, in a digital credential platform server along with anassociated issued digital credential and/or the associated user.Physical user cues may include, for example, facial expressions, userreactions and/or noises made by the user during testing/simulations,user body language, eye movement, and any other user behavior orreaction detectable via cameras and external sensors. Additionally oralternatively, various types of user biometric data also may becollected during the testing, simulation, and/or monitoring processesperformed on the user. Such biometric data may include, for instance,the user's temperature, heartrate, blood pressure, respiration, skinconductivity, and brainwave activity, and/or any known types ofbiometric data that may collected during testing, credentialing, and/ormonitoring processes.

As discussed in more detail below, the user's physical cues and/orbiometric data may be collected and saved within a digital credentialplatform server, and associated with the user, one or more particulardigital credentials, and/or with the particulartesting/simulation/monitoring processes during which the data wasoriginally detected. Once collected, the data may be used toauthenticate the testing, simulation, and/or monitoring processes, toconfirm the user's identity and to prevent errors or fraudulentactivities by users. The data may be saved with the user's digitalcredential and/or into a separate portfolio of evidence, which may beavailable to the user for review, and also may be provided upon requestto potential employers for review during a review or hiring process.Such evidence data also may be applied to updated credentialingrequirements, so that in some cases a user may simply resubmit theirevidence portfolio instead of being required to recertify their digitalcredential when the test or credentialing standards are updated. Incertain embodiments, the user's physical cues and/or biometric data alsomay be analyzed to determine the user's emotional states and reactionsduring the testing, simulation, and/or monitoring. Additionally oralternatively, the physical cues and biometric data may be detected forseveral users and analyzed collectively to provide feedback regardingthe digital credential testing processes, simulations, monitoring,physical testing environments, etc.

Referring now to FIGS. 17A-17B, examples are shown illustrating facialrecognition and analysis functionality that may be performed inconnection with a user testing/credentialing process (live orsimulation), or with user on-the-job credentialing or monitoringprocesses. In this example, one or more cameras may be configured tocapture the user's facial features and expressions at different pointsduring the testing/credentialing/monitoring processes. For testsperformed within a simulation lab-type physical testing environment, anumber of designated cameras may capture not only the user's face butalso the user's body from several different angles. Thus, certainphysical testing environments may be capable not only of capturingfacial images of the user, but also detecting detailed facialexpressions at different times during the test/simulation, andpotentially eye movement patterns, gestures, body language, and anyother non-verbal and non-written user expression data.

In other embodiments, such as for certain on-the-job credentialing ormonitoring systems, or for formal testing/credentialing whensophisticated high-tech physical testing environments are not used, thephysical cue data and/or biometrics data collected may be limited by thecameras and sensors available. In some cases, a laptop camera or webcaminstalled at the user's workstation may be use to capture facial imagesand/or to recognize facial expressions at different times during thetesting/monitoring. However, such cameras may or may not have theresolution and image capture capabilities to perform advanced facialexpression monitoring, eye movement, and/or body language detection. Inother examples, such as on-the-job credentialing and monitoringscenarios, facial images might only be detectable using lower-qualitysecurity cameras or the like that are configured to monitor an entirefloor or workspace. In such examples, the facial images may be still beuseful for certain purposes (e.g., confirmation of user identification),but potential may be unsuitable for facial expression analysis, eyemovement analysis, and the like.

Additionally or alternatively, physical testing environments (e.g.,simulation labs) and/or workstation or workplace monitoring systems mayinclude various biometric sensors configured to detect biometric data ofthe user at different times during the test/simulation. As noted above,such biometric data may include the user's temperature, heartrate, bloodpressure, respiration, skin conductivity, and brainwave activity, and/orany known types of biometric data. Thus, the biometric metric may bedetected and captured via a combination of external sensors, wearablesensors, and/or implanted sensors in some cases. For on-the-jobcredentialing and monitoring, mobile wearable sensors such as heartratemonitors, step trackers, and the like, may be used when more advancedwearable sensors (e.g., blood pressure, respiration, skin conductivity,brainwave activity, etc.) are not practical.

Referring now to FIG. 18, a flow diagram is shown illustrating anexpress process of collecting physical cue data and/or biometric datafor a user during a user testing, credentialing, or monitoringprocesses, and using the physical cue and biometrics to authenticate theuser's identity and the associated data. The process shown in thisexample may be implemented within any of the testing/credentialingsystems, simulators, workstation or workplace monitoring systems, andthe like described herein. In step 1801, a testing, credentialing,and/or monitoring system such as those described above may execute atest, simulation, or user monitoring process for a particular user inconnection with a digital credential that the user is seeking or hasalready obtained. The particular types of tests may include, forexample, live simulations and/or virtual or augmented realitysimulations executed within a physical testing environment 700. In otherexamples, the testing in step 1801 may correspond to an on-the-jobcredentialing system that monitors and evaluates a user's workplacetasks and activities, or to a post-credentialing user monitoring systemconfigured to determine whether the user is using their previouslyissued digital credentials. In step 1802, during any of these testing,simulation, or monitoring processes, one or more of the user monitoringdevices described above, including cameras, microphones, motion sensors,tracking devices, and/or user biometrics sensors, may capture physicalcues from the user and/or biometric data of the user during the testing,simulation, or monitoring processes. Such physical cues may includeparticular facial expressions, user reactions and/or noises made by theuser during testing/simulations/monitoring, as well as user bodylanguage and eye movements. In step 1803, the physical cue and userbiometric data may be encapsulated and transmitted to the transmitted tothe platform server 1410. In other embodiments, certain systems (e.g.,1421-1423) may retain and store user's physical cues and biometrics datalocally, rather than the evidence data being stored in a centralrepository. Regardless of storage location, the physical cues andbiometrics data of the user may be associated with particular testquestions and/or particular time stamps during a testing or simulation.Additionally, when the data is transmitted, it may be compressed andedited as needed, and/or encrypted in order to assure data security anduser privacy.

In some embodiments, the platform server 1410 may use the physical cuesand/or biometrics data collected for the user as part of anauthentication process in step 1804. For example, during anytesting/credentialing process (e.g., written testing, computer-basedtesting, simulation lab testing, etc.) the user's facial images,physical cues, and/or biometrics may be compared against previouslystored corresponding data (e.g., user images, physical cue patterns,biometrics, etc.) in order to verify that the correct user is taking thetest/simulation. Additionally, the user's physical cues and biometricsmay provide an additional level of authentication, by comparing theobserved physical cues and biometrics at particular times during thetest or simulation to expected physical cues and biometrics, based onwhat is happening during the test or simulation at that particular time.For instance, a simulation may be designed to present a challenging andstressful situation to the user at a particular timestamp or within asequence of tasks the user is performed. In step 1804, the server maycompare the user's observed physical cues and biometrics to the physicalcues and biometrics that would be expected for the challenging andstressful situation, in order to confirm that the data is valid and/orthat the user did not expect this situation in advance (e.g., indicatingcheating). In step 1805, the platform serving 1410 having validated theuser's identity and the authenticity of the user's physical cues andbiometrics, may store the testing, credentialing, monitoring data in thedigital credential data store as valid data. In some embodiments, theimage data, facial cues, and/or biometrics data also may be retained andstored by the platform server for future analysis.

In some embodiments, the data relating to the user's physical cues andbiometrics collected during a test, simulation, or during on-the-jobmonitoring, may be further evaluated to identify the user's emotionalstates at different times. For instance, certain simulations may bespecifically designed to invoke certain emotional states (e.g., anger,boredom, frustration, surprise, etc.), and the user's level ofperformance while experiencing those emotional states may beparticularly important for certain testing/credentialing processes. Inthese examples and other cases, either exhibiting or not exhibitingparticular emotion states may be an eligibility requirement for acredential receiver to obtain certain types of digital credentials.Thus, the data collected during the test, simulation, or monitoring instep 1801 may be used not only for user identification/authentication,but also may be analyzed to (1) determine the user's emotional state atdifferent times during the test, simulation, or monitoring, (2) comparethat emotional state to an expected emotional state based on what theuser is experiencing, and (3) evaluate the user's reactions, levels ofskills performance during different emotional states.

Additionally, in some embodiments, the physical cues, biometrics data,and/or emotional states detected for multiple users may be aggregatedfor the same tests, simulations, monitoring environments, etc. Theaggregated data for tests may be used to revise current tests andsimulations, design new tests and simulations, and for training usershow to respond to particular scenarios and situations (e.g., workplaceaccidents).

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

1. A digital credential platform server comprising: a processing unitcomprising one or more processors; and memory coupled with and readableby the processing unit and storing therein a set of instructions which,when executed by the processing unit, causes the digital credentialplatform server to: receive input data identifying a first digitalcredential receiver; retrieve one or more generated digital credentials,including a first digital credential, issued to the first digitalcredential receiver; determine a plurality of activities associated withone or more eligibility criteria for the first digital credential;initiate monitoring of a physical environment associated with the firstcredential receiver, using a plurality of sensors; detect, using theplurality of sensors, a plurality of user actions performed by the firstcredential receiver, over a time period, within the physicalenvironment; determine a plurality of activities performed by the firstcredential receiver over the time period, based on an analysis of thedetected plurality of user actions; for each particular activity of theplurality of activities associated with the eligibility criteria for thefirst digital credential, compare the plurality of activities performedby the first credential receiver over the time period, to the particularactivity; and output data corresponding to the comparison of each of theplurality of activities associated with the eligibility criteria for thefirst digital credential, to the plurality of activities performed bythe first credential receiver over the time period.
 2. The digitalcredential platform server of claim 1, the memory storing thereinfurther instructions which, when executed by the processing unit, causesthe digital credential platform server to: determine an expiration timefor the first digital credential issued to the first digital credentialreceiver, based on the comparison the plurality of activities associatedwith the first digital credential to the plurality of activitiesperformed by the first credential receiver over the time period.
 3. Thedigital credential platform server of claim 1, the memory storingtherein further instructions which, when executed by the processingunit, causes the digital credential platform server to: determine avaluation of the first digital credential, based on the comparison theplurality of activities associated with the first digital credential tothe plurality of activities performed by the first credential receiverover the time period.
 4. The digital credential platform server of claim1, wherein comparing the plurality of activities performed by the firstcredential receiver to the particular activity comprises: retrieving athreshold number associated with the particular activity; analyzing thedetected plurality of user actions to determine a number of theparticular activity performed by the credential receiver during the timeperiod; and comparing the threshold number associated with theparticular activity, to the number of the particular activity performedby the credential receiver during the time period.
 5. The digitalcredential platform server of claim 1, wherein comparing the pluralityof activities performed by the first credential receiver to theparticular activity comprises: determining an error rate of thecredential receiver when performing the particular activity during thetime period; and comparing the error rate of the credential receiverwhen performing the particular activity, to a threshold error rateassociated with the particular activity.
 6. The digital credentialplatform server of claim 1, wherein detecting the plurality of useractions performed by the first credential receiver, over a time period,within the physical environment comprises: activating the plurality ofsensors, and configuring the plurality of sensors to detect theplurality of user actions, wherein the configuration the plurality ofsensors is based on the plurality of activities associated with theeligibility criteria for the first digital credential.
 7. The digitalcredential platform server of claim 1, wherein the plurality of sensorscomprises a plurality of video recording devices, and wherein theplurality of user actions are detected based on video data captured by acombination of the plurality of video recording devices.
 8. The digitalcredential platform server of claim 1, wherein the plurality of sensorscomprises software-based sensors executing as a background process on acomputer terminal operated by the credential receiver, thesoftware-based sensors configured to detect at least one of (a)keystrokes entered via a keyboard connected to the computer terminal,(b) mouse events performed via a mouse connected to the computerterminal, and (c) touch events performed via a touchscreen displayconnected to the computer terminal.
 9. A method of analyzing digitalcredentials, comprising: receiving, by a digital credential platformserver, input data identifying a first digital credential receiver;retrieving, by the digital credential platform server, one or moregenerated digital credentials, including a first digital credential,issued to the first digital credential receiver; determining, by thedigital credential platform server, a plurality of activities associatedwith one or more eligibility criteria for the first digital credential;monitoring, by the digital credential platform server, of a physicalenvironment associated with the first credential receiver, using aplurality of sensors; detecting, by the digital credential platformserver, using the plurality of sensors, a plurality of user actionsperformed by the first credential receiver, over a time period, withinthe physical environment; determining, by the digital credentialplatform server, a plurality of activities performed by the firstcredential receiver over the time period, based on an analysis of thedetected plurality of user actions; for each particular activity of theplurality of activities associated with the eligibility criteria for thefirst digital credential, comparing, by the digital credential platformserver, the plurality of activities performed by the first credentialreceiver over the time period, to the particular activity; andoutputting, by the digital credential platform server, datacorresponding to the comparison of each of the plurality of activitiesassociated with the eligibility criteria for the first digitalcredential, to the plurality of activities performed by the firstcredential receiver over the time period.
 10. The method of analyzingdigital credentials of claim 9, further comprising: determining anexpiration time for the first digital credential issued to the firstdigital credential receiver, based on the comparison the plurality ofactivities associated with the first digital credential to the pluralityof activities performed by the first credential receiver over the timeperiod.
 11. The method of analyzing digital credentials of claim 9,further comprising: determining a valuation of the first digitalcredential, based on the comparison the plurality of activitiesassociated with the first digital credential to the plurality ofactivities performed by the first credential receiver over the timeperiod.
 12. The method of analyzing digital credentials of claim 9,wherein comparing the plurality of activities performed by the firstcredential receiver to the particular activity comprises: retrieving athreshold number associated with the particular activity; analyzing thedetected plurality of user actions to determine a number of theparticular activity performed by the credential receiver during the timeperiod; and comparing the threshold number associated with theparticular activity, to the number of the particular activity performedby the credential receiver during the time period.
 13. The method ofanalyzing digital credentials of claim 9, wherein comparing theplurality of activities performed by the first credential receiver tothe particular activity comprises: determining an error rate of thecredential receiver when performing the particular activity during thetime period; and comparing the error rate of the credential receiverwhen performing the particular activity, to a threshold error rateassociated with the particular activity.
 14. The method of analyzingdigital credentials of claim 9, wherein detecting the plurality of useractions performed by the first credential receiver, over a time period,within the physical environment comprises: activating the plurality ofsensors, and configuring the plurality of sensors to detect theplurality of user actions, wherein the configuration the plurality ofsensors is based on the plurality of activities associated with theeligibility criteria for the first digital credential.
 15. The method ofanalyzing digital credentials of claim 9, wherein the plurality ofsensors comprises a plurality of video recording devices, and whereinthe plurality of user actions are detected based on video data capturedby a combination of the plurality of video recording devices.
 16. Themethod of analyzing digital credentials of claim 9, wherein theplurality of sensors comprises software-based sensors executing as abackground process on a computer terminal operated by the credentialreceiver, the software-based sensors configured to detect at least oneof (a) keystrokes entered via a keyboard connected to the computerterminal, (b) mouse events performed via a mouse connected to thecomputer terminal, and (c) touch events performed via a touchscreendisplay connected to the computer terminal.
 17. A non-transitorycomputer-readable medium, having instructions stored therein, which whenexecuted by a computing device cause the computing device to perform aset of operations comprising: receiving input data identifying a firstdigital credential receiver; retrieving one or more generated digitalcredentials, including a first digital credential, issued to the firstdigital credential receiver; determining a plurality of activitiesassociated with one or more eligibility criteria for the first digitalcredential; monitoring of a physical environment associated with thefirst credential receiver, using a plurality of sensors; detecting usingthe plurality of sensors, a plurality of user actions performed by thefirst credential receiver, over a time period, within the physicalenvironment; determining a plurality of activities performed by thefirst credential receiver over the time period, based on an analysis ofthe detected plurality of user actions; for each particular activity ofthe plurality of activities associated with the eligibility criteria forthe first digital credential, comparing the plurality of activitiesperformed by the first credential receiver over the time period, to theparticular activity; and outputting data corresponding to the comparisonof each of the plurality of activities associated with the eligibilitycriteria for the first digital credential, to the plurality ofactivities performed by the first credential receiver over the timeperiod.
 18. The non-transitory computer-readable medium of claim 17, theinstructions causing the computing device to perform further operationscomprising: determining an expiration time for the first digitalcredential issued to the first digital credential receiver, based on thecomparison the plurality of activities associated with the first digitalcredential to the plurality of activities performed by the firstcredential receiver over the time period.
 19. The non-transitorycomputer-readable medium of claim 17, the instructions causing thecomputing device to perform further operations comprising: determining avaluation of the first digital credential, based on the comparison theplurality of activities associated with the first digital credential tothe plurality of activities performed by the first credential receiverover the time period.
 20. The non-transitory computer-readable medium ofclaim 17, wherein detecting the plurality of user actions performed bythe first credential receiver, over a time period, within the physicalenvironment comprises: activating the plurality of sensors, andconfiguring the plurality of sensors to detect the plurality of useractions, wherein the configuration the plurality of sensors is based onthe plurality of activities associated with the eligibility criteria forthe first digital credential.